Reason of having big RAM buffer is lowering frequency of IndexWriter flushes and (subsequently) lowering frequency of index merge events, and (subsequently) merging of a few larger files takes less time... especially if RAM Buffer is intelligent enough (and big enough) to deal with 100 concurrent updates of existing document without 100-times flushing to disk of 100 document versions.
I posted here thread related; I had 1:5 timing for Update:Merge (5 minutes merge, and 1 minute update) with default SOLR settings (32Mb buffer). I increased buffer to 8Gb on Master, and it triggered significant indexing performance boost... -Fuad http://www.linkedin.com/in/liferay > -----Original Message----- > From: Mark Miller [mailto:markrmil...@gmail.com] > Sent: October-23-09 3:03 PM > To: solr-user@lucene.apache.org > Subject: Re: Too many open files > > I wouldn't use a RAM buffer of a gig - 32-100 is generally a good number. > > Fuad Efendi wrote: > > I was partially wrong; this is what Mike McCandless (Lucene-in-Action, 2nd > > edition) explained at Manning forum: > > > > mergeFactor of 1000 means you will have up to 1000 segments at each level. > > A level 0 segment means it was flushed directly by IndexWriter. > > After you have 1000 such segments, they are merged into a single level 1 > > segment. > > Once you have 1000 level 1 segments, they are merged into a single level 2 > > segment, etc. > > So, depending on how many docs you add to your index, you'll could have > > 1000s of segments w/ mergeFactor=1000. > > > > http://www.manning-sandbox.com/thread.jspa?threadID=33784&tstart=0 > > > > > > So, in case of mergeFactor=100 you may have (theoretically) 1000 segments, > > 10-20 files each (depending on schema)... > > > > > > mergeFactor=10 is default setting... ramBufferSizeMB=1024 means that you > > need at least double Java heap, but you have -Xmx1024m... > > > > > > -Fuad > > > > > > > >> I am getting too many open files error. > >> > >> Usually I test on a server that has 4GB RAM and assigned 1GB for > >> tomcat(set JAVA_OPTS=-Xms256m -Xmx1024m), ulimit -n is 256 for this > >> server and has following setting for SolrConfig.xml > >> > >> > >> > >> <useCompoundFile>true</useCompoundFile> > >> > >> <ramBufferSizeMB>1024</ramBufferSizeMB> > >> > >> <mergeFactor>100</mergeFactor> > >> > >> <maxMergeDocs>2147483647</maxMergeDocs> > >> > >> <maxFieldLength>10000</maxFieldLength> > >> > >> > > > > > > > > > -- > - Mark > > http://www.lucidimagination.com > >
