Most users do not set the direct memory setting. mlockall is set, but does the server allow it? You would see an error on startup if it didn't. Did you change the vm swapiness on the server?
-- Ivan On Wed, Jul 16, 2014 at 2:40 AM, Pedro Jerónimo <[email protected]> wrote: > > *Java: *java version "1.7.0_55" > *ElasticSearch: *1.2.1 > > > On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote: > >> What ES, and what Java version is this? >> >> Jörg >> >> >> On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo <[email protected]> >> wrote: >> >>> I have a cluster of 2 ES machines with a lot of indexing, not so much >>> searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on >>> each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it >>> logs1) is running logstash as well, with 2gb heap. The master node is logs1 >>> and the other instance is logs2. I start the cluster and every things looks >>> fine, but after a while (1-3 days) I get the following error on logs1: >>> >>> [2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] >>> exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 >>> => /XX.XXX.XX.XX:9300]], closing connection >>> java.lang.OutOfMemoryError: Direct buffer memory >>> ...Stack Trace... >>> >>> And then the cluster is no longer connected and if I try to restart >>> logs2, I get the same error above for logs1 and this one for logs2: >>> >>> [2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner] >>> failed to send join request to master [[Keen Marlow][9a7FIRpBSrKQcdcV_ >>> sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a, >>> master=true}], reason [org.elasticsearch.transport.RemoteTransportException: >>> [Keen Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join]; >>> org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross >>> Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] >>> disconnected] >>> >>> Is there any memory configuration I should tune up a bit? I'm kind of >>> new to ElasticSearch so I'd love some help! :). >>> >>> Thanks! >>> >>> Pedro >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD6Zp%2BHMvFTaUQ3L5FXV7sKaR4MYSVOVQa%3DZzfsBrB7Lg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
