Thanks for the the help, Doug.
-Matt Zytaruk
Doug Cutting wrote:
Matt Zytaruk wrote:
Well, if we want to fetch pages from N different sites, ideally we
should be able to have N threads running, without any of them having
to wait. I guess ideally what the fetcher should probably do is
instead of waiting, put the url it was trying to fetch back into the
queue to be tried later on, and grab a different one.
The fetcher used to do this, and it ended up with huge queues. We
capped the size of the queues, and dropped urls when their queue was
full. But the fetcher still spent an age at the end, mostly idle,
with a single thread emptying its queue. And there were some bugs in
the queue synchronization that caused things to sometimes hang, but no
one could ever figure out why.
So the current fetcher's strategy is to, instead of queuing urls in
order to drop them later, drop them now. And instead of queuing urls
in order to wait later, wait now. It makes things a lot simpler. In
the end the performance is similar, but you can see the cost of
crawling big sites immediately, rather than only later. In either
case you need to choose to drop things or run slowly.
I'm not so sure that accesses to each host are spread evenly
throughout the list, because the fetch list I was doing had tens of
thousands of different hosts and I was still getting a large amount
of threads trying to access the same host at the same time, even with
only 50 threads. Although maybe I'm wrong and that is how it would
act if the hosts were spread evenly throughout, I'm not sure, it just
seems like a lot.
They're not spread exactly evenly, but randomly, which can be a bit
lumpy. What percentage of urls in the fetch list are from a host that
is exceeding max delays? If it is near 2%, or that host is slower
than average, then you'll probably have issues with 50 threads.
Doug
