Which version of ES are you using? Versions before 1.2 have a bug that caused merge throttling to throttle far more than requested such that you couldn't get any faster than ~8 MB / sec. See https://github.com/elasticsearch/elasticsearch/issues/6018
Tiered merge policy is best. Mike McCandless http://blog.mikemccandless.com On Mon, Aug 25, 2014 at 1:08 PM, Chris Decker <[email protected]> wrote: > All, > > I’m looking for advice on how to reduce the number of segments for my > indices because in my use case (log analysis), quick searches are more > important than real-time access to data. I've turned many of the "knobs" > available within ES, and read many blog postings, ES documentation, etc., > but still feel like there is room for important. > > Specific questions I have: > 1. How can I increase the current merge rate? According to Elastic HQ, my > merge rate is 6 MB/s (according to Elastic HQ). I know I don't have SSDs, > but with 15k drives it seems like I should be able to get better rates. I > tried increasing indices.store.throttle.max_bytes_per_sec from the default > of 20mb to 40mb in my templates, but I didn't see a noticeable change in > disk IOps or the merge rate the next day. Did I do something incorrectly? > I'm going to experiment with setting it overall > with index.store.throttle.max_bytes_per_sec and removing it from my > templates. > 2. Should I move away from the default merge policy, or stick with the > default ("tiered")? > > Any advice you have is much appreciated; additional details on my > situation are below. > > ---- > > - I generate 2 indices per day - “high” and “low”. I usually end up with > ~ 450 segments for my ‘high’ index (see attached), and another ~ 200 > segments for my ‘low’ index, which I then optimize once I roll-over to the > next day’s indices. > - 4 ES servers (soon to be 8). > — Each server has: > 12 Xeon cores running at 2.3 GHz > 15k drives > 128 GB of RAM > 68 GB used for OS / file system machine > 60 GB used by 2 JVMs > - Index ~ 750 GB per day; 1.5 TB if you include the replicas > - Relevant configs: > TEMPLATE: > "index.refresh_interval" : "60s", > "index.number_of_replicas" : "1", > "index.number_of_shards" : "4", > "index.merge.policy.max_merged_segment" : "50g", > "index.merge.policy.segments_per_tier" : "5", > "index.merge.policy.max_merge_at_once" : “5”, > "indices.store.throttle.max_bytes_per_sec" : "40mb". > > ELASTICSEARCH.YML: > indices.memory.index_buffer_size: 30% > > > > Thanks in advance!, > Chris > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smReAtdSsxEnJzXH%2BAWxSv6G5_-iQWUdbhzu3__rH4LsTNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
