So far so good...up to over 1 million pages rewritten use Andrzej fix! I was waiting to post until it gets all the way done, but so far I have been able to get through my thick list of segments!
I can't wait to see some search speeds now! :) You guys rock. Jason ----- Original Message ----- From: "Andrzej Bialecki" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Wednesday, October 06, 2004 1:42 PM Subject: Re: [Nutch-dev] SegmentMergeTool 2 > Doug Cutting wrote: > > Andrzej Bialecki wrote: > > > >> Ah, but now it finally starts making sense... This points to a real > >> deficiency in SegmentMergeTool (or perhaps a wrong assumption). When > >> reading segment data I assumed that if the segment is truncated, all > >> data files (content, parse_text, fetcher, etc) will be truncated to > >> the same index position. Apparently this is not always the case. > > > > > > This is not guaranteed. They're generally truncated to the last buffer > > flush. We do not flush after each document fetched. That could be > > added, but it might slow things somewhat. > > No, it's ok the way it is - I was just lazy when coding the original > version of the tool. Using Jason as my guinea pig ;-) I believe I > arrived at a patch that works around this and other problems. > > I assumed that Michael C. will commit shortly my patches to > LocalFileSystem, if not, I can do this myself, so long as we know who's > doing what... > > -- > Best regards, > Andrzej Bialecki > > ------------------------------------------------- > Software Architect, System Integration Specialist > CEN/ISSS EC Workshop, ECIMF project chair > EU FP6 E-Commerce Expert/Evaluator > ------------------------------------------------- > FreeBSD developer (http://www.freebsd.org) > > > > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal > Use IT products in your business? Tell us what you think of them. Give us > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
