What is your soft commit interval? That'll cause I/O as well. How much physical RAM and how much is dedicated to _all_ the JVMs on a machine? One cause here is that Lucene uses MMapDirectory which can be starved for OS memory if you use too much JVM, my rule of thumb is that _at least_ half of the physical memory should be reserved for the OS.
Your transaction logs should fluctuate but even out. By that I mean they should increase in size but every hard commit should truncate some of them so I wouldn't expect them to grow indefinitely. One strategy is to put your tlogs on a separate drive exactly to reduce contention. You could disable them too at a cost of risking your data. That might be a quick experiment you could run though, disable tlogs and see what that changes. Of course I'd do this on my test system ;). But yeah, Solr will use a lot of I/O in the scenario you are outlining I'm afraid. Best, Erick On Wed, Jul 5, 2017 at 8:08 AM, Antonio De Miguel <deveto...@gmail.com> wrote: > thanks Markus! > > We already have SSD. > > About changing topology.... we probed yesterday with 10 shards, but system > goes more inconsistent than with the current topology (5x10). I dont know > why... too many traffic perhaps? > > About merge factor.. we set default configuration for some days... but when > a merge occurs system overload. We probed with mergefactor of 4 to improbe > query times and trying to have smaller merges. > > 2017-07-05 16:51 GMT+02:00 Markus Jelsma <markus.jel...@openindex.io>: > >> Try mergeFactor of 10 (default) which should be fine in most cases. If you >> got an extreme case, either create more shards and consider better hardware >> (SSD's) >> >> -----Original message----- >> > From:Antonio De Miguel <deveto...@gmail.com> >> > Sent: Wednesday 5th July 2017 16:48 >> > To: solr-user@lucene.apache.org >> > Subject: Re: High disk write usage >> > >> > Thnaks a lot alessandro! >> > >> > Yes, we have very big physical dedicated machines, with a topology of 5 >> > shards and10 replicas each shard. >> > >> > >> > 1. transaction log files are increasing but not with this rate >> > >> > 2. we 've probed with values between 300 and 2000 MB... without any >> > visible results >> > >> > 3. We don't use those features >> > >> > 4. No. >> > >> > 5. I've probed with low and high mergefacors and i think that is the >> point. >> > >> > With low merge factor (over 4) we 've high write disk rate as i said >> > previously >> > >> > with merge factor of 20, writing disk rate is decreasing, but now, with >> > high qps rates (over 1000 qps) system is overloaded. >> > >> > i think that's the expected behaviour :( >> > >> > >> > >> > >> > 2017-07-05 15:49 GMT+02:00 alessandro.benedetti <a.benede...@sease.io>: >> > >> > > Point 2 was the ram Buffer size : >> > > >> > > *ramBufferSizeMB* sets the amount of RAM that may be used by Lucene >> > > indexing for buffering added documents and deletions before >> they >> > > are >> > > flushed to the Directory. >> > > maxBufferedDocs sets a limit on the number of documents >> buffered >> > > before flushing. >> > > If both ramBufferSizeMB and maxBufferedDocs is set, then >> > > Lucene will flush based on whichever limit is hit first. >> > > >> > > <ramBufferSizeMB>100</ramBufferSizeMB> >> > > <maxBufferedDocs>1000</maxBufferedDocs> >> > > >> > > >> > > >> > > >> > > ----- >> > > --------------- >> > > Alessandro Benedetti >> > > Search Consultant, R&D Software Engineer, Director >> > > Sease Ltd. - www.sease.io >> > > -- >> > > View this message in context: http://lucene.472066.n3. >> > > nabble.com/High-disk-write-usage-tp4344356p4344386.html >> > > Sent from the Solr - User mailing list archive at Nabble.com. >> > > >> > >>
