Mike, Thanks for the response.
I'm running ES 1.2.1. It appears the issue that you reported / corrected was included with ES 1.2.0. *Any other ideas / suggestions? *Were the settings that I posted sane? Thanks!, Chris On Monday, August 25, 2014 1:52:46 PM UTC-4, Michael McCandless wrote: > > Which version of ES are you using? Versions before 1.2 have a bug that > caused merge throttling to throttle far more than requested such that you > couldn't get any faster than ~8 MB / sec. See > https://github.com/elasticsearch/elasticsearch/issues/6018 > > Tiered merge policy is best. > > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, Aug 25, 2014 at 1:08 PM, Chris Decker <[email protected] > <javascript:>> wrote: > >> All, >> >> I’m looking for advice on how to reduce the number of segments for my >> indices because in my use case (log analysis), quick searches are more >> important than real-time access to data. I've turned many of the "knobs" >> available within ES, and read many blog postings, ES documentation, etc., >> but still feel like there is room for important. >> >> Specific questions I have: >> 1. How can I increase the current merge rate? According to Elastic HQ, >> my merge rate is 6 MB/s (according to Elastic HQ). I know I don't have >> SSDs, but with 15k drives it seems like I should be able to get better >> rates. I tried increasing indices.store.throttle.max_bytes_per_sec from >> the default of 20mb to 40mb in my templates, but I didn't see a noticeable >> change in disk IOps or the merge rate the next day. Did I do something >> incorrectly? I'm going to experiment with setting it overall >> with index.store.throttle.max_bytes_per_sec and removing it from my >> templates. >> 2. Should I move away from the default merge policy, or stick with the >> default ("tiered")? >> >> Any advice you have is much appreciated; additional details on my >> situation are below. >> >> ---- >> >> - I generate 2 indices per day - “high” and “low”. I usually end up with >> ~ 450 segments for my ‘high’ index (see attached), and another ~ 200 >> segments for my ‘low’ index, which I then optimize once I roll-over to the >> next day’s indices. >> - 4 ES servers (soon to be 8). >> — Each server has: >> 12 Xeon cores running at 2.3 GHz >> 15k drives >> 128 GB of RAM >> 68 GB used for OS / file system machine >> 60 GB used by 2 JVMs >> - Index ~ 750 GB per day; 1.5 TB if you include the replicas >> - Relevant configs: >> TEMPLATE: >> "index.refresh_interval" : "60s", >> "index.number_of_replicas" : "1", >> "index.number_of_shards" : "4", >> "index.merge.policy.max_merged_segment" : "50g", >> "index.merge.policy.segments_per_tier" : "5", >> "index.merge.policy.max_merge_at_once" : “5”, >> "indices.store.throttle.max_bytes_per_sec" : "40mb". >> >> ELASTICSEARCH.YML: >> indices.memory.index_buffer_size: 30% >> >> >> >> Thanks in advance!, >> Chris >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46ecc658-502f-46c7-b2b9-db9fd0e9f58f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
