Hello, I'm using Nutch's fetcher and want to make use of its ability to perform multiple smaller fetches and create separate segments. Once I have multiple segments I would like to merge them and remove duplicate URLs (leaving just one reference to a URL, say the most recent one). I know this is possible with SegmentMergeTool that's in CVS now. However, this SegmentMergeTool doesn't suit me, as it creates a 'Nutch/Lucene index' while merging segments. I have no use for this index (I'm only using Nutch's fetcher now) and would like to avoid creating it (I'm working with a large number of URLs).
So, my question is: is it possible to merge and de-duplicate multiple segments without creating the index? Thanks, Otis ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
