I've been using nutch for a little while now, and the new release is great. I'm hoping someone can help me with what I'm trying to do.
One of the sites I crawl is basically an archive for a mailing list. So there's lots of data that never changes, and then there are new pages every day. I'm not entirely clear on how this recrawling thing works. Is there a way I can use Nutch to just crawl those new pages, and ignore all the old ones that are pretty much static forever that I've already crawled? Any help would be greatly appreciated. Thanks, Ann __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
