What is a safe number to delay between page requests from the same host? I want to crawl as much information as possible in the shortest amount of time, but I also don't want to hurt the server i'm crawling.... What do you guys use? I am using 5 seconds right now.
------------------------------------------------------- This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes! http://ads.osdn.com/?ad_ids93&alloc_id281&op=click _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
