You can add a "Crawl-delay: xx" instruction, where "xx" is a delay value between successive crawler accesses [http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html]
If I do this for Nutch bot, will it work? Does it get Crawl-delay instructions and update the interval? Any other similar ones for threads.... -- View this message in context: http://www.nabble.com/crawl-delay-and-nutch-tf3528305.html#a9845743 Sent from the Nutch - User mailing list archive at Nabble.com.
