Andrzej, --- Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote: > > Andrzej (sorry for forgetting the 'z' earlier), > > No problem - most foreigners are taken by surprise with my name... > :-) You're not alone :) > > Thanks for the answer. That makes sense and that's what I thought > was > > the issue - the Lucene index is just used for quick lookups. > > How about de-duplication - if there are duplicate URL references in > > multiple segments, which reference wins? The most recent one? > > Yes. I've noticed what looks like a strange behaviour, and I'm not sure if it's a bug or by design. Scenario: 0. consider a fetch list with a single URL 1. URL was fetched successfully in one segment 2. at some later time, in a second segment, fetching of the same URL failed (e.g. host unknown error) 3. the two segments are merged with SegmentMergeTool 4. looking at the Metadata (ParseData.getMetadata()) for the resulting segment I get data from the first successful fetch. Step 4 is what surprises me. Since the last attempt to fetch the URL failed, I thought that would be the data in the final, merged segment. Instead, I got the data about the last successful fetch. With the current code I can't tell that the last fetch attempt failed, or at least I think I can't. Looking at the db/segments/... files I see that even after I merge the two segments, the resulting files are the exact same size as after step 1. Is there a way to get failure information from one of the 4 data files? Thanks, Otis > However, your question started me thinking... perhaps there is too > much > work put into indexing, deduplication, re-indexing etc. I'm trying > now > to write a variant of SegmentMerge which avoids most of these steps > by > using certain properties of Lucene indexes. This could result in > radical > speed improvements. Stay tuned. > > -- > Best regards, > Andrzej Bialecki > > ------------------------------------------------- > Software Architect, System Integration Specialist > CEN/ISSS EC Workshop, ECIMF project chair > EU FP6 E-Commerce Expert/Evaluator > ------------------------------------------------- > FreeBSD developer (http://www.freebsd.org) > > > > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on > ITManagersJournal > Use IT products in your business? Tell us what you think of them. > Give us > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find > out more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
