UPDATE: I have 100,000,000 new documents in 24 hours, including possible updates OR possibly adding same document several times. I have two segments now (30Gb total), and network is overloaded (I use web crawler to generate documents). I never had more than 25,000,000 within a month before...
I read that high mergeFactor improves performance of updates; however, it didn't work (it delays all merges... commit/optimize took similar timing). High ramBufferSizeMB does the job. [Fuad Efendi] >Looks like I temporarily solved the problem with not-so-obvious settings: [Fuad Efendi] >ramBufferSizeMB=8192 [Fuad Efendi] >mergeFactor=10 > Never tried profiling; > 3000-5000 docs per second if SOLR is not busy with segment merge; > > During segment merge 99% CPU, no disk swap; I can't suspect I/O... > > During document updates (small batches 100-1000 docs) only 5-15% CPU > > constant rate 5:1 is very suspicious... > >> In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM >> Buffer >> Flash / Segment Merge per 1 minute of (heavy) batch document updates.
