Andrzej (sorry for forgetting the 'z' earlier), Thanks for the answer. That makes sense and that's what I thought was the issue - the Lucene index is just used for quick lookups. How about de-duplication - if there are duplicate URL references in multiple segments, which reference wins? The most recent one?
Thanks, Otis P.S. Sivi - neka onda onaj code ostane kako je. --- Andrzej Bialecki <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > > --- [EMAIL PROTECTED] wrote: > > > > > >>Hello, > >> > >>I'm using Nutch's fetcher and want to make use of its ability to > >>perform multiple smaller fetches and create separate segments. > Once > >>I > >>have multiple segments I would like to merge them and remove > >>duplicate > >>URLs (leaving just one reference to a URL, say the most recent > one). > >>I > >>know this is possible with SegmentMergeTool that's in CVS now. > >>However, this SegmentMergeTool doesn't suit me, as it creates a > >>'Nutch/Lucene index' while merging segments. I have no use for > this > >>index (I'm only using Nutch's fetcher now) and would like to avoid > >>creating it (I'm working with a large number of URLs). > >> > >>So, my question is: is it possible to merge and de-duplicate > multiple > >>segments without creating the index? > > The issue with the de-dup process is that you need a fast and > efficient > lookup structure to determine if a given url is unique, otherwise you > > can't determine if it's already stored in another segment. Since > Nutch > already uses Lucene indexes it makes sense to use Lucene to create > such > a unique index... > > Of course, if you insist, you can create such an index by emitting > triplets (url, docid, segment) to a text file and to use some Unix > scripting magic, or even an SQL DB - but I bet in the end it will be > much slower and consume much more space... > ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
