There's a bunch of kernel/OS tweaks you can apply as well. eg noatime, nodiratime if you mount the ES dir on it's own, or some of these http://namhuy.net/1563/how-to-tweak-and-optimize-ssd-for-ubuntu-linux-mint.html Then indices.store.throttle.max_bytes_per_sec might be worth looking at, you can increase that from the 20mb default.
I know there'd be a bunch of people that would be interested in your setup and the tweaks you've done if you're interested in putting up a blog post/gist with it. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 18 March 2014 07:43, <[email protected]> wrote: > We haven't tried Java 8 or changing garbage collection yet -- I've heard > mixed results on GC. We're using SSD storage on Azure, and have a fairly > tweaked out config. I was thinking about turning off any kind of analysis > in the template to get it to scale... > > On Sunday, March 16, 2014 7:37:09 PM UTC-7, Mark Walkom wrote: >> >> We are getting ~15K/s with 12 data + 3 master nodes, latest version of >> java and ES. >> >> Some things to try would be; >> >> - Java 8 >> - G1GC >> - SSD storage >> - tweaking various index and pool caches >> - optimising your data inputs, mappings etc >> >> We're trialling the first two of those on our dev cluster, but it doesn't >> do much traffic so I cannot empirically comment on it's capabilities at the >> levels you're after as yet. >> >> What does your current setup (ie infrastructure) look like now? >> >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 17 March 2014 13:17, <[email protected]> wrote: >> >>> We've gotten a cluster up to 40K/sec using roughly 40 nodes. We're >>> going to switch over to using dedicated master nodes as well. This is all >>> in AWS. >>> >>> -Justin >>> >>> On Sunday, March 16, 2014 9:14:43 PM UTC-4, Janet Sullivan wrote: >>>> >>>> I currently have a proof of concept cluster handling about 12000 >>>> msgs/sec. If a certain project kicks off, I would need to scale 10+ >>>> times. Is anyone successfully running ES at over 150k/sec? What kind of >>>> shard layout are you using, how many data nodes, etc? >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/f6c889f0-5288-4aef-b2c8-007b9272b94a% >>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f6c889f0-5288-4aef-b2c8-007b9272b94a%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/1f430f4b-369f-4db7-bb00-669455ca63f3%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/1f430f4b-369f-4db7-bb00-669455ca63f3%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZrQTpgew8ttC2SHKyZv_dLoyp5%2BGToHC8mr4NmH8SosQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
