Hi, Toke, Did you try MapReduce with solr? I think it should be a good fit for your use case.
On Tue, Jun 16, 2015 at 5:02 AM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > Shenghua(Daniel) Wan <wansheng...@gmail.com> wrote: > > Actually, I am currently interested in how to boost merging/optimizing > > performance of single solr instance. > > We have the same challenge (we build static 900GB shards one at a time and > the final optimization takes 8 hours with only 1 CPU core at 100%). I know > that there is code for detecting SSDs, which should make merging faster (by > running more merges in parallel?), but I am afraid that optimize (a single > merge) is always single threaded. > > It seems to me that at least some of the different files making up a > segment could be created in parallel, but I do not know how hard it would > be to do so. > > - Toke Eskildsen > -- Regards, Shenghua (Daniel) Wan
