[EMAIL PROTECTED] wrote:
However, I am nervous about fetching 10M pages at a time because of
the pain
that would result if a fetch cycle failed, and would prefer to fetch
smaller
1M-page segments if there is a way to combine the segments
afterwards. Is
there a way to do that?
BTW, I do understand that multi-segment indexes can be merged, so I'm
only
asking about the segment data themselves.
In Nutch 0.7 there was a SegmentMergeTool, which performed exactly this
function. It hasn't been ported yet to the map-reduce branch; it's on my
medium-term TODO list, unless someone beats me to it ...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
