Hello,
I'm using Nutch to crawl (without indexing) the website imdb and I get the following error for a large number of URLs :
fetch of http://www.imdb.com/title/tt0437543/ failed with: org.apache.nutch.protocol.RetryLater: Exceeded http.max.delays: retry later.
Even by increasing the value of http.max.delays to 500 in nutch-default.xml (making the crawl very very slow), I can't fetch the page (it is not in my webdb at the end of the process)...
What's wrong ? How to best configure the crawl in this case ? Thanks for your great help !!!
Fabrice
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
