Hello, I think this question/problem/bug was missed due to a bad subject line I used earlier. Here it is again, with a better subject.
> > > How about de-duplication - if there are duplicate URL references > in > > > multiple segments, which reference wins? The most recent one? > > > > Yes. Below is the description of the problem (Nutch 0.5). > I've noticed what looks like a strange behaviour, and I'm not sure if > it's a bug or by design. > > Scenario: > > 0. consider a fetch list with a single URL > 1. URL was fetched successfully in one segment > 2. at some later time, in a second segment, fetching of the same URL > failed (e.g. host unknown error) > 3. the two segments are merged with SegmentMergeTool > 4. looking at the Metadata (ParseData.getMetadata()) for the > resulting segment I get data from the _first_ fetch - the successful > one. > > Step 4 is what surprises me. Since the last attempt to fetch the URL > failed, I thought that failure status would be the data in the final, > merged segment. > Instead, I got the data about the last successful fetch. > > With the current code I can't tell that the last fetch attempt > failed, > or at least I think I can't. Looking at the db/segments/... files I > see that even after I merge the two segments, the resulting files are > the exact same size as after step 1. > > Is there a way to get failure information from one of the 4 data > files? > > Thanks, > Otis ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
