@Mark, I'm using this cluster only for logstash, so I have 1 new index p/day, each index has between 30k and 100k documents. I use curator in a crontab to close old logs older than X days so I don't have them all loaded into memory at the same time. Right now I have 8 indices opened with a total of ~500k documents occupying ~4.5Gb of disk space. I read a bit about OOM with ElasticSearch and I couldn't figure out my problem yet. I followed ES docs memory tips:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html And I see that mlockall is set to true on both instances. @Ivan, yes, ES and Logstash are the only processes running on the servers. No, I didn't set direct memory values. Should I? If so, which sort of values and how do you determine those? It's been a while since I worked with JVMs so my knowledge on Java Memory stuff is kind of rusty :). Thanks for the help! On Wednesday, July 16, 2014 5:40:27 AM UTC+2, Ivan Brusic wrote: > > Direct memory is off heap memory. Are elasticsearch and logstash the only > processes on those servers? Did you set an explicit direct memory value? > > -- > Ivan > On Jul 15, 2014 3:46 PM, "Mark Walkom" <[email protected] > <javascript:>> wrote: > >> How much data do you have in ES, index count and total size of all >> indexes? >> >> OutOfMemoryError means you ran out of heap, which could mean a few things. >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] <javascript:> >> web: www.campaignmonitor.com >> >> >> On 15 July 2014 22:33, Pedro Jerónimo <[email protected] <javascript:> >> > wrote: >> >>> I have a cluster of 2 ES machines with a lot of indexing, not so much >>> searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on >>> each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it >>> logs1) is running logstash as well, with 2gb heap. The master node is logs1 >>> and the other instance is logs2. I start the cluster and every things looks >>> fine, but after a while (1-3 days) I get the following error on logs1: >>> >>> [2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] >>> exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 >>> => /XX.XXX.XX.XX:9300]], closing connection >>> java.lang.OutOfMemoryError: Direct buffer memory >>> ...Stack Trace... >>> >>> And then the cluster is no longer connected and if I try to restart >>> logs2, I get the same error above for logs1 and this one for logs2: >>> >>> [2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner] >>> failed to send join request to master [[Keen >>> Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a, >>> >>> master=true}], reason >>> [org.elasticsearch.transport.RemoteTransportException: [Keen >>> Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join]; >>> org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross >>> Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected] >>> >>> Is there any memory configuration I should tune up a bit? I'm kind of >>> new to ElasticSearch so I'd love some help! :). >>> >>> Thanks! >>> >>> Pedro >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bb843355-0c29-4ec1-803a-9149b89fc36a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
