Has anyone written an API that can merge thousands of segments? The current segment merge tool cannot handle this much data as there just isn't enough RAM available on the box. So, I was wondering if there was a better, incremental way to handle this.
Currently I have 1 segment for each domain that was crawled and I want to merge them all into several large segments. So, if anyone has any pointers I would appreciate it. Has anyone else attempted to keep segments at this granularity? This doesn't seem to work so well. <briggs /> "Concious decisions by concious minds are what make reality real"
