I don't know the implementation details, but in the extreme scenario you described, where number of threads is set to 100, but only 1 is active, because there is only 1 host to crawl, there is not much you can do - you don't want more than 1 thread hitting the same host.
Are you saying the current implementation results in sleeping threads, when in fact they could be fetching pages? Otis --- Stefan Groschupf <[EMAIL PROTECTED]> wrote: > Hi developers. > > I would love to discuss the thread management of the fetcher, may be > there is room for improvements. > The fetcher start as much threads as setuped in the configuration > file > for example 100 threads. > The problem that i see is that in case the fetchlist contains many > urls > of the same host a set of threads sleep and is not used to fetch. > Or do I oversee something in the code? > So in case we start n threads and actually m threads sleep since the > host is blocked there are only n-m thread working. > > Wouldn't it be better to have every-time n threads working? > I see 2 possible solutions to solve that problem: > > 1.) We change the way we handle delayed urls. For example write > blocked > urls back to a queue and the thread try the next url. > 2.) We monitor and handle the number of threads actually working. For > > example a thread that starts to sleep announce that to a handler. The > > handler could start a new thread for the time the other thread is > sleeping. > > I personal think the first solution would be better however this > require some more detailed technical discussion. > > Any comments? > > Greetings, > Stefan > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real > users. > Discover which products truly live up to the hype. Start reading now. > > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
