I upgraded "master" to 1.4-dev from trunk 3 days ago BTW such performance broke my "commodity hardware", most probably network card... can't SSH to check stats; need to check onsite what happened...
-----Original Message----- From: Grant Ingersoll Sent: August-13-09 4:20 PM To: solr-user@lucene.apache.org Subject: Re: Performance Tuning: segment_merge:index_update=5:1 (timing) BTW, what version of Solr are you on? On Aug 13, 2009, at 1:43 PM, Fuad Efendi wrote: > UPDATE: > > I have 100,000,000 new documents in 24 hours, including possible > updates OR > possibly adding same document several times. I have two segments now > (30Gb > total), and network is overloaded (I use web crawler to generate > documents). > I never had more than 25,000,000 within a month before... > > I read that high mergeFactor improves performance of updates; > however, it > didn't work (it delays all merges... commit/optimize took similar > timing). > High ramBufferSizeMB does the job. > > > [Fuad Efendi] >Looks like I temporarily solved the problem with > not-so-obvious settings: > [Fuad Efendi] >ramBufferSizeMB=8192 > [Fuad Efendi] >mergeFactor=10 > > > >> Never tried profiling; >> 3000-5000 docs per second if SOLR is not busy with segment merge; >> >> During segment merge 99% CPU, no disk swap; I can't suspect I/O... >> >> During document updates (small batches 100-1000 docs) only 5-15% CPU >> >> constant rate 5:1 is very suspicious... >> >>> In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM >>> Buffer >>> Flash / Segment Merge per 1 minute of (heavy) batch document >>> updates. > > > -------------------------- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
