Hi all,
I have been working on Fetcher2 code lately and I came across this
particular code (in FetchItemQueue.getFetchItem) that I didn't quite
understand:
public FetchItem getFetchItem() {
...
long last = endTime.get() + (maxThreads > 1 ? crawlDelay : minCrawlDelay);
...
}
Now, the 'default' politeness behaviour should be 1 thread per host
and delaying n seconds between successive requests to that host,
right? But, won't this code wait only minCrawlDelay(which, by default,
is 0) if maxThreads == 1.
I also did not understand why there is a maxThread check at all. Each
individual thread should wait crawl delay before making another
request to the same host. Am I missing something here?
--
Doğacan Güney
