Another thing to try, is reducing the maxThreadCount for ConcurrentMergeScheduler.
It defaults to 3, which I think is too high -- we should change this default to 1 (I'll open a Lucene issue). Mike On Thu, Nov 12, 2009 at 6:30 PM, Jerome L Quinn <jlqu...@us.ibm.com> wrote: > > Hi, everyone, this is a problem I've had for quite a while, > and have basically avoided optimizing because of it. However, > eventually we will get to the point where we must delete as > well as add docs continuously. > > I have a Solr 1.3 index with ~4M docs at around 90G. This is a single > instance running inside tomcat 6, so no replication. Merge factor is the > default 10. ramBufferSizeMB is 32. maxWarmingSearchers=4. > autoCommit is set at 3 sec. > > We continually push new data into the index, at somewhere between 1-10 docs > every 10 sec or so. Solr is running on a quad-core 3.0GHz server. > under IBM java 1.6. The index is sitting on a local 15K scsi disk. > There's nothing > else of substance running on the box. > > Optimizing the index takes about 65 min. > > As long as I'm not optimizing, search and indexing times are satisfactory. > > When I start the optimize, I see massive problems with timeouts pushing new > docs > into the index, and search times balloon. A typical search while > optimizing takes > about 1 min instead of a few seconds. > > Can anyone offer me help with fixing the problem? > > Thanks, > Jerry Quinn
