In this case, a host is an IP address.
I've thought about this more, and wonder if perhaps this should be
switched so that host name are blocked from simultaneous fetching rather
than IP addresses. I recently spoke with Carlos Castillo, author of the
WIRE crawler (http://www.cwr.cl/projects/WIRE/) and it blocks hosts by
name, not IP. What do others think?
I think IP adresses are much better.
We are splitting one of our big websites to many subdomains.
We think our customers could realize our urls in this case better.
I think there are other people out there, doing the same.
All subdomains are handled by the same host / IP.
All big search engines seems to think that each subdomain is a single
host and stress each subdomain as single host.
In our case that's bad. We could not handle one request for each
subdomain at the same time. So we have to answer with 503 much
searchengines according to the system load. Specially it is very bad, if
all the big search engines crawl our subdomains at same time.
So using IP adresses to limit the access to one host is much better.
Matthias
