Hi - there's some setting fetcher queue exceptions something that will dump a queue if it had too many exceptions.
-----Original message----- > From:Adam Estrada <[email protected]> > Sent: Thursday 17th July 2014 16:07 > To: [email protected] > Subject: Ignoring errors in crawl > > All, > > I am coming across a few pages that are not responsive at all which is > causing Nutch to #failwhale before finishing the current crawl. I have > increased http.timeout and it still crashes. How can I get Nutch to > skip over unresponsive URLs that are causing the entire thing to bail? > > Thanks, > Adam >
