On Thu, 2005-10-13 at 16:42 -0400, Rod Taylor wrote: > On Thu, 2005-10-13 at 13:35 -0700, AJ Chen wrote: > > I try to fetch as fast as it can by using more threads on a large fetch > > list. But, the fetcher starts download at speed much lower than the full > > bandwidth allows. And the start download speed varies a lot from run to run, > > 200kb/s to 1200kb/s on my DSL line. This variation also happens on T1 line > > that I just tested. > > Could someone share experience on how to make fetcher use the full > > bandwidth? We know the speed drops gradually during a long fetch run. But, > > can the fetch achieve the highest speed allowed by the bandwidth when fetch > > starts?
> 4000 threads with a local caching DNS server seems to be enough to fill > the pipe though Your also limited by the number of servers you are connecting out to since Nutch will by default limit itself to asking for a single page at a time from a single server. -- Rod Taylor <[EMAIL PROTECTED]>
