A good place to begin would be to look at the log file generated by Hadoop, which would be in "log/" directory. The file automatically truncates each day, but that shouldn't be a problem. You could parse that after each crawl, just taking the link and excluding all the other information. ----- Original Message ---- From: djames <[EMAIL PROTECTED]> To: [email protected] Sent: Thursday, March 8, 2007 8:10:11 AM Subject: external host link logging
Hello, I'm working with nutch since 2 month now, and i'm very happy to see that this project is so powerfull!!!!! I need to crawl only a set of given website, so i set the parameter db.ignore.external.links to false and it works perfectly. But now i need to create a log file with the list of all links parsed or fetched leading to external host for a human validation and reinjection in the crawl db. I don't now how to begin??? Could someone help me please Thanks a lot -- View this message in context: http://www.nabble.com/external-host-link-logging-tf3369106.html#a9374136 Sent from the Nutch - User mailing list archive at Nabble.com.
