Lets assume that I am crawling wikipedia.org with depth 1 and topN 1. After it finishes crawling if I rerun that command and after finishes again and again. What happens? Does Nutch skips previous fetched pages or try to crawl same pages again?
-- View this message in context: http://lucene.472066.n3.nabble.com/Does-Nutch-Checks-Whether-A-Page-crawled-before-or-not-tp4049564.html Sent from the Nutch - User mailing list archive at Nabble.com.
