Andrzej Bialecki wrote:
Ah, but now it finally starts making sense... This points to a real deficiency in SegmentMergeTool (or perhaps a wrong assumption). When reading segment data I assumed that if the segment is truncated, all data files (content, parse_text, fetcher, etc) will be truncated to the same index position. Apparently this is not always the case.
This is not guaranteed. They're generally truncated to the last buffer flush. We do not flush after each document fetched. That could be added, but it might slow things somewhat.
No, it's ok the way it is - I was just lazy when coding the original version of the tool. Using Jason as my guinea pig ;-) I believe I arrived at a patch that works around this and other problems.
I assumed that Michael C. will commit shortly my patches to LocalFileSystem, if not, I can do this myself, so long as we know who's doing what...
-- Best regards, Andrzej Bialecki
------------------------------------------------- Software Architect, System Integration Specialist CEN/ISSS EC Workshop, ECIMF project chair EU FP6 E-Commerce Expert/Evaluator ------------------------------------------------- FreeBSD developer (http://www.freebsd.org)
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
