Where does Nutch stores that information? 2013/3/21 Markus Jelsma-2 [via Lucene] < [email protected]>
> Nutch selects records that are eligible for fetch. It's either due to a > transient failure or if the fetch interval has been expired. This means > that failed fetches due to network issues are refetched within 24 hours. > Successfully fetched pages are only refetched if the current time exceeds > the previously fetchTime + interval. > > > > -----Original message----- > > > From:kamaci <[hidden > > email]<http://user/SendEmail.jtp?type=node&node=4049568&i=0>> > > > Sent: Wed 20-Mar-2013 23:46 > > To: [hidden email]<http://user/SendEmail.jtp?type=node&node=4049568&i=1> > > Subject: Does Nutch Checks Whether A Page crawled before or not > > > > Lets assume that I am crawling wikipedia.org with depth 1 and topN 1. > After > > it finishes crawling if I rerun that command and after finishes again > and > > again. What happens? Does Nutch skips previous fetched pages or try to > crawl > > same pages again? > > > > > > > > -- > > View this message in context: > http://lucene.472066.n3.nabble.com/Does-Nutch-Checks-Whether-A-Page-crawled-before-or-not-tp4049564.html > > Sent from the Nutch - User mailing list archive at Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://lucene.472066.n3.nabble.com/Does-Nutch-Checks-Whether-A-Page-crawled-before-or-not-tp4049564p4049568.html > To unsubscribe from Does Nutch Checks Whether A Page crawled before or > not, click > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4049564&code=ZnVya2Fua2FtYWNpQGdtYWlsLmNvbXw0MDQ5NTY0fDEyODM4MDc0Mg==> > . > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://lucene.472066.n3.nabble.com/Does-Nutch-Checks-Whether-A-Page-crawled-before-or-not-tp4049564p4049569.html Sent from the Nutch - User mailing list archive at Nabble.com.
