Hi, I have in my nutch-site.xml the value fetcher.max.crawl.delay = -1.
When I try to fetch a site with a robots.txt with a Crawl Delay, it doesn't work. If I put fetcher.max.crawl.delay = 10000, it works. I use Nutch 1.2, but according to the changelog, nothing has been changed about that since then. Is this a Nutch bug or I misused something ? Another thing, in hadoop.log, the pages which couldn't be fetched are still marked as "fetching", is this normal ? Shouldn't they be marked as "dropped" or something ? Thanks.
