Matt Zytaruk wrote:
Indeed, that does work, although that ends up slowing down the fetch a
fair amount because a lot of threads end up idle, waiting, and I was
hoping to avoid that slowdown if possible.
What should these threads be doing?
If you have a site with N pages to fetch, and you want to fetch them all
politely, then it will take at least fetcher.server.delay*N to fetch
them all. The fetch list is sorted by the hash of the url, so accesses
to each host should be spread fairly evenly through the list.
Capping the number of pages per host (generate.max.per.host) will help,
or, if you know the webmasters in question, you can consider increasing
fetcher.threads.per.host.
Doug
