BTW, what version of Solr are you on?
On Aug 13, 2009, at 1:43 PM, Fuad Efendi wrote:
UPDATE:
I have 100,000,000 new documents in 24 hours, including possible
updates OR
possibly adding same document several times. I have two segments now
(30Gb
total), and network is overloaded (I use web crawler to generate
documents).
I never had more than 25,000,000 within a month before...
I read that high mergeFactor improves performance of updates;
however, it
didn't work (it delays all merges... commit/optimize took similar
timing).
High ramBufferSizeMB does the job.
[Fuad Efendi] >Looks like I temporarily solved the problem with
not-so-obvious settings:
[Fuad Efendi] >ramBufferSizeMB=8192
[Fuad Efendi] >mergeFactor=10
Never tried profiling;
3000-5000 docs per second if SOLR is not busy with segment merge;
During segment merge 99% CPU, no disk swap; I can't suspect I/O...
During document updates (small batches 100-1000 docs) only 5-15% CPU
constant rate 5:1 is very suspicious...
In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM
Buffer
Flash / Segment Merge per 1 minute of (heavy) batch document
updates.
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
