Hi Erik! thanks for your response! Our soft commit is 5 seconds. Why generates I/0 a softcommit? first notice.
We have enough physical RAM to store full collection and 16Gb for each JVM. The collection is relatively small. I've tried (for testing purposes) disabling transactionlog (commenting <updateLog>)... but cluster does not go up. I'll try writing into separated drive, nice idea... 2017-07-05 18:04 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > What is your soft commit interval? That'll cause I/O as well. > > How much physical RAM and how much is dedicated to _all_ the JVMs on a > machine? One cause here is that Lucene uses MMapDirectory which can be > starved for OS memory if you use too much JVM, my rule of thumb is > that _at least_ half of the physical memory should be reserved for the > OS. > > Your transaction logs should fluctuate but even out. By that I mean > they should increase in size but every hard commit should truncate > some of them so I wouldn't expect them to grow indefinitely. > > One strategy is to put your tlogs on a separate drive exactly to > reduce contention. You could disable them too at a cost of risking > your data. That might be a quick experiment you could run though, > disable tlogs and see what that changes. Of course I'd do this on my > test system ;). > > But yeah, Solr will use a lot of I/O in the scenario you are outlining > I'm afraid. > > Best, > Erick > > On Wed, Jul 5, 2017 at 8:08 AM, Antonio De Miguel <deveto...@gmail.com> > wrote: > > thanks Markus! > > > > We already have SSD. > > > > About changing topology.... we probed yesterday with 10 shards, but > system > > goes more inconsistent than with the current topology (5x10). I dont know > > why... too many traffic perhaps? > > > > About merge factor.. we set default configuration for some days... but > when > > a merge occurs system overload. We probed with mergefactor of 4 to > improbe > > query times and trying to have smaller merges. > > > > 2017-07-05 16:51 GMT+02:00 Markus Jelsma <markus.jel...@openindex.io>: > > > >> Try mergeFactor of 10 (default) which should be fine in most cases. If > you > >> got an extreme case, either create more shards and consider better > hardware > >> (SSD's) > >> > >> -----Original message----- > >> > From:Antonio De Miguel <deveto...@gmail.com> > >> > Sent: Wednesday 5th July 2017 16:48 > >> > To: solr-user@lucene.apache.org > >> > Subject: Re: High disk write usage > >> > > >> > Thnaks a lot alessandro! > >> > > >> > Yes, we have very big physical dedicated machines, with a topology of > 5 > >> > shards and10 replicas each shard. > >> > > >> > > >> > 1. transaction log files are increasing but not with this rate > >> > > >> > 2. we 've probed with values between 300 and 2000 MB... without any > >> > visible results > >> > > >> > 3. We don't use those features > >> > > >> > 4. No. > >> > > >> > 5. I've probed with low and high mergefacors and i think that is the > >> point. > >> > > >> > With low merge factor (over 4) we 've high write disk rate as i said > >> > previously > >> > > >> > with merge factor of 20, writing disk rate is decreasing, but now, > with > >> > high qps rates (over 1000 qps) system is overloaded. > >> > > >> > i think that's the expected behaviour :( > >> > > >> > > >> > > >> > > >> > 2017-07-05 15:49 GMT+02:00 alessandro.benedetti <a.benede...@sease.io > >: > >> > > >> > > Point 2 was the ram Buffer size : > >> > > > >> > > *ramBufferSizeMB* sets the amount of RAM that may be used by Lucene > >> > > indexing for buffering added documents and deletions before > >> they > >> > > are > >> > > flushed to the Directory. > >> > > maxBufferedDocs sets a limit on the number of documents > >> buffered > >> > > before flushing. > >> > > If both ramBufferSizeMB and maxBufferedDocs is set, then > >> > > Lucene will flush based on whichever limit is hit first. > >> > > > >> > > <ramBufferSizeMB>100</ramBufferSizeMB> > >> > > <maxBufferedDocs>1000</maxBufferedDocs> > >> > > > >> > > > >> > > > >> > > > >> > > ----- > >> > > --------------- > >> > > Alessandro Benedetti > >> > > Search Consultant, R&D Software Engineer, Director > >> > > Sease Ltd. - www.sease.io > >> > > -- > >> > > View this message in context: http://lucene.472066.n3. > >> > > nabble.com/High-disk-write-usage-tp4344356p4344386.html > >> > > Sent from the Solr - User mailing list archive at Nabble.com. > >> > > > >> > > >> >
