Hi all,
I have been working on Fetcher2 code lately and I came across this
particular code (in FetchItemQueue.getFetchItem) that I didn't quite
understand:
public FetchItem getFetchItem() {
...
long last = endTime.get() + (maxThreads > 1 ? crawlDelay : minCrawlDelay);
...
}
Now, the 'default' politeness behaviour should be 1 thread per host
and delaying n seconds between successive requests to that host,
right? But, won't this code wait only minCrawlDelay(which, by default,
is 0) if maxThreads == 1.
I also did not understand why there is a maxThread check at all. Each
individual thread should wait crawl delay before making another
request to the same host. Am I missing something here?
--
Doğacan Güney
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
