You should drop your heap to 31GB, over that you don't compress java pointers and GC will suffer.
1. That will help as it reduces document size. 2. Definitely, if you don't need near-realtime access then open it up, we run at 60sec but could probably go to 2 or more minutes. 3. This could be risky, doubly so given you are only running two nodes. To elaborate on point 3, and on general note, you should really run 3 or more nodes incrementing up to odd numbers (3,5,7,9....). The reason for this is that is that it helps to prevent split brain across your cluster. You should use disable bloom filtering on your indexes as it will give you a bit of a boost, Elasticsearch curator can handle that for you. However you are probably reaching the limits of your cluster, 2 billion docs is a fair bit of data. What versions of ES and java are you on? What amount of data (GB) is it? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 3 October 2014 00:04, Mathias Adler <[email protected]> wrote: > Hi All, > I'm quite new to ES and using it as part of the ELK stack. I'm running a > two node cluster where each node got 24core and 64GB RAM (32 allocated to > Java), We have an index rate @ ~3000/s and total document count per 24h @ > 200miljon (and storing 10days). > When mem usage gets close to 80% I start having search problems in Kibana > and get all kinds of exception and all gets better when mem usage gets > lower again. > So, one way is of course to scale out, but my question is, what mem tuning > can be done and will it make a big difference? > 1, Drop unused data fields in logstash already, will that make any > difference? > 2, Reduce index.refresh_interval, will that reduce mem usage? > 3, Set replicas to zero and get all primary shards spread over both nodes, > will that impact mem usage? > What else can be done to lower mem usage? > > Anyone out there having the same type of load or higher (document count > and index rate), how does your set up look like? > > Br > Mathias > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/2ba446f8-6017-4863-b462-a075b60780f2%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/2ba446f8-6017-4863-b462-a075b60780f2%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bs6A2z7pxvxeN7YYxkp%2BDph851nLKejjG6WC9jEasWcw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
