All, I am coming across a few pages that are not responsive at all which is causing Nutch to #failwhale before finishing the current crawl. I have increased http.timeout and it still crashes. How can I get Nutch to skip over unresponsive URLs that are causing the entire thing to bail?
Thanks, Adam
