I don't know the implementation details, but in the extreme scenario you described, where number of threads is set to 100, but only 1 is active, because there is only 1 host to crawl, there is not much you can do - you don't want more than 1 thread hitting the same host.
Sure, in case you do a intranet crawl you wish to have only one thread.
Are you saying the current implementation results in sleeping threads, when in fact they could be fetching pages?
Right, I had some scenarios where many pages was on one host but there was still other hosts and since a lot of threads was sleeping the fetcher was running only with half power.
Stefan
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
