So most other crawlers use the hostname, not the ip. That's good to
know. google and yahoo, Yes. The others I am not sure.
Perhaps a dynamic property would help. If the elapsed time of the previous request is some fraction of the delay then we might lessen the delay. Similarly, if it is greater or if we get 503s, then we might increase it. For example, if the fraction were .5 and the delay is 2 seconds, then sites which respond faster than a second would get their delay decreased, and sites which respond in more than a second or that return 503 would have their delay increased. Do you think this would be effective with your site?
Adjusting the amount of downloads dynamically according to the response time should be great. But where is the advantage doing this per unique name? If there is no real reason to do so, I would do it dynamically per IP or second level domain, but not per sub domain. Matthias
