To give my input, i didn't use bandwidth throttle, i just tried different thread counts and still have (had issues) issues.
I got best performance when i excluded https (no parser - just gave an exception error) and ftp urls. Even then after a while it would slowwwwwwwwwwwwww downnnnnnnnnnnn So i'm glad to see focus on one scheduler and will be more than happy to test it out in a big run once cvs stabalizes a bit. -byron --- Stefan Groschupf <[EMAIL PROTECTED]> wrote: > Doug, > > cool, thanks a lot! > ;-) > > > - I removed the RequestScheduler fetcher > implementation, as I only > > have time to maintain a single fetcher. If > someone would like, then > > they can port RequestScheduler to the new APIs. > Fetcher.java should > > now correctly support robots.txt and be polite. > > > Do we need a new RequestScheduler? > I definitely figure out that the instability was > related to the > bandwidth limitation mechanism, but it was no fixed > with my posted > suggestion. > I notice the fetcher had the same problem. > > > > Stefan > > > --------------------------------------------------------------- > open technology: http://www.media-style.com > open source: http://www.weta-group.net > open discussion: http://www.text-mining.org > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new > InstallShield X. > From Windows to Linux, servers to mobile, > InstallShield X is the one > installation-authoring solution that does it all. > Learn more and > evaluate today! > http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers ------------------------------------------------------- This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one installation-authoring solution that does it all. Learn more and evaluate today! http://www.installshield.com/Dev2Dev/0504 _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
