Direct memory is off heap memory. Are elasticsearch and logstash the only processes on those servers? Did you set an explicit direct memory value?
-- Ivan On Jul 15, 2014 3:46 PM, "Mark Walkom" <[email protected]> wrote: > How much data do you have in ES, index count and total size of all indexes? > > OutOfMemoryError means you ran out of heap, which could mean a few things. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] > web: www.campaignmonitor.com > > > On 15 July 2014 22:33, Pedro Jerónimo <[email protected]> wrote: > >> I have a cluster of 2 ES machines with a lot of indexing, not so much >> searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on >> each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1) >> is running logstash as well, with 2gb heap. The master node is logs1 and >> the other instance is logs2. I start the cluster and every things looks >> fine, but after a while (1-3 days) I get the following error on logs1: >> >> [2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] >> exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 >> => /XX.XXX.XX.XX:9300]], closing connection >> java.lang.OutOfMemoryError: Direct buffer memory >> ...Stack Trace... >> >> And then the cluster is no longer connected and if I try to restart >> logs2, I get the same error above for logs1 and this one for logs2: >> >> [2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner] >> failed to send join request to master [[Keen >> Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a, >> master=true}], reason >> [org.elasticsearch.transport.RemoteTransportException: [Keen >> Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join]; >> org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross >> Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected] >> >> Is there any memory configuration I should tune up a bit? I'm kind of new >> to ElasticSearch so I'd love some help! :). >> >> Thanks! >> >> Pedro >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDXTAG7%3Dz66Ghs7H3hfSTmLuEprUjoKK-4XEv_tSbENBA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
