Indeed, that does work, although that ends up slowing down the fetch a
fair amount because a lot of threads end up idle, waiting, and I was
hoping to avoid that slowdown if possible.
-Matt
Doug Cutting wrote:
Please try simply increasing http.max.delays, perhaps to 100 or more.
Probably we should increase the default, since with partitioned fetch
lists this is much more common.
Doug
Matt Zytaruk wrote:
Hello all, just have a question on the fetcher in 0.8 dev.
A big problem I've been noticing with fetching is the number of max
delays exceeded errors. I recently did a fetch of about 600,000 pages
and 1 in 6 of those was never downloaded because of max delays
exceeded. This seems like an excessive amount, so I decided to look
into the code to see if I could find out how the url's are sorted for
fetching, so that maybe I could improve this to spread out the
attempts to fetch from each host. I saw that the urllist is
partitioned by host at one point, but I gather that is just so that
all the urls from one host are sent to the same server for fetching,
to ensure politeness? I couldn't track down any other sorting. Am I
missing something here? Is there any other way that the fetcher could
be improved so that I don't get so many max delays exceeded errors?
Thanks in advance.
-Matt Zytaruk
