Ah, but now it finally starts making sense... This points to a real deficiency in SegmentMergeTool (or perhaps a wrong assumption). When reading segment data I assumed that if the segment is truncated, all data files (content, parse_text, fetcher, etc) will be truncated to the same index position. Apparently this is not always the case.
This is not guaranteed. They're generally truncated to the last buffer flush. We do not flush after each document fetched. That could be added, but it might slow things somewhat.
Doug
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
