On Thu, 2005-10-13 at 13:35 -0700, AJ Chen wrote: > I try to fetch as fast as it can by using more threads on a large fetch > list. But, the fetcher starts download at speed much lower than the full > bandwidth allows. And the start download speed varies a lot from run to run, > 200kb/s to 1200kb/s on my DSL line. This variation also happens on T1 line > that I just tested. > Could someone share experience on how to make fetcher use the full > bandwidth? We know the speed drops gradually during a long fetch run. But, > can the fetch achieve the highest speed allowed by the bandwidth when fetch > starts?
I found that for high bandwidth (50Mbits and above) DNS seems to be a limiting factor. 4000 threads with a local caching DNS server seems to be enough to fill the pipe though -- Rod Taylor <[EMAIL PROTECTED]> ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
