Go with 1 thread per host. For my small area of internet where I fetch my pages, almost all hosts starts refusing requests on 3+ threads, some of them, even at 1+.
Bandwidth-wise, if you go with higher value on fetcher.threads.per.host, your fetcher will have hard time connecting and the target server will save on bandwidth in fact ;) -----Original Message----- From: Raymond Creel [mailto:[EMAIL PROTECTED] Sent: Monday, July 25, 2005 4:00 PM To: [email protected] Subject: fetch bandwidth settings I have read that you don't want to make more than 1 or 2 requests per second to the same host, or else you will start adversely affecting their bandwidth. Is this a good rule of thumb? Along those lines, what would be the best values to put in the nutch config file to maximize speed of fetching without hammering the site? I'm thinking something like this: fetcher.server.delay: 1.0 fetcher.threads.per.host: 2 thanks, raymond ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs
