Hello Team, I am running an experimental cluster of 5 0.8-dev (2006-03-18) fetchers, and intending to crawl 100 Million pages.
I understand that a balance must be achieved between having too few and too many
segments on the searchers. I intend to - run 5 searchers - each responsible for 20M pages - with 10M pages per segment (2 segments per searcher) However, I am nervous about fetching 10M pages at a time because of the pain that would result if a fetch cycle failed, and would prefer to fetch smaller 1M-page segments if there is a way to combine the segments afterwards. Is there a way to do that? BTW, I do understand that multi-segment indexes can be merged, so I'm only asking about the segment data themselves. Many thanks, Monu Ogbe
