On Mon, Dec 20, 2004 at 11:55:09AM +0100, Stefan Groschupf wrote: > >I don't know the implementation details, but in the extreme scenario > >you described, where number of threads is set to 100, but only 1 is > >active, because there is only 1 host to crawl, there is not much you > >can do - you don't want more than 1 thread hitting the same host. > Sure, in case you do a intranet crawl you wish to have only one thread. > > > > >Are you saying the current implementation results in sleeping threads, > >when in fact they could be fetching pages? > > > Right, I had some scenarios where many pages was on one host but there > was still other hosts and since a lot of threads was sleeping the > fetcher was running only with half power. > > Stefan
There was this old fetcher RequestScheduler.java. It is more sophisticated in design, but had issues and was abondoned due to lack of maintenance. You may want to check it again. John ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
