Is there any way to prevent ES from blowing up just by selecting too much data? This is my biggest concern. Is it because the bootstrap.mlockall is on, so we give ES/JVM a specified amount of memory and thats all that node will receive? If we turned that off and had gobs more swap available for ES, would it not blow up, but just be real slow?
On Mon, May 5, 2014 at 4:12 PM, Mark Walkom <[email protected]>wrote: > Then you need more nodes, more heap on existing nodes or less data. > You've reached the limit of what your current cluster can handle, that is > why this is happening. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] > web: www.campaignmonitor.com > > > On 6 May 2014 09:11, Nate Fox <[email protected]> wrote: > >> I have 11 nodes. 3 are dedicated masters and the other 8 are data nodes. >> On May 5, 2014 4:03 PM, "[email protected]" <[email protected]> >> wrote: >> >>> You have only two nodes it seems. Adding nodes may help. >>> >>> Beside data nodes that do the heavy work, set up 3 master eligible nodes >>> (data-less nodes, with reasonable smaller heap size for cluster state and >>> mappings). Set the other data nodes to non-eligible for becoming master. >>> >>> Jörg >>> >>> >>> On Mon, May 5, 2014 at 9:34 PM, Nate Fox <[email protected]> wrote: >>> >>>> We're using ES 1.1.0 for central logging storage/searching. When we use >>>> Kibana and search a month's worth of data, our cluster becomes >>>> unresponsive. By unresponsive I mean that many nodes will respond >>>> immediately to a 'curl localhost:9200' but a couple will not. This leads to >>>> any cluster metrics not being available when quering the master and we're >>>> unable to set any cluster-level settings. >>>> >>>> We're getting a these types of errors in the logs: >>>> [2014-05-05 19:10:50,763][WARN ][transport.netty ] [Leap-Frog] >>>> exception caught on transport layer [[id: 0x4b074069, / >>>> 10.6.10.211:57563 => /10.6.10.148:9300]], closing connection >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> The cluster seems to never recover either - and that is my biggest >>>> concern. So my questions are: >>>> 1. Is it normal for the entire cluster to just close up shop because a >>>> couple nodes are unresponsive? I thought the field data circuit breaker >>>> would fix this, but maybe this is a different problem. >>>> 2. How to best get ES to recover from this scenario? I dont really want >>>> to restart just the two nodes, as we have >1Tb of data on each node, but >>>> issuing a disable_allocation fails because it cannot write to all nodes in >>>> the cluster >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/2fb4e427-cf95-4882-bd87-728fbfef10dd%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/2fb4e427-cf95-4882-bd87-728fbfef10dd%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/pNgeukzPL3A/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE2EXjGifcjQkvhE1NmeEnHUJO%3Dr-iB7E%3DLzY-jxz%2BAAw%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE2EXjGifcjQkvhE1NmeEnHUJO%3Dr-iB7E%3DLzY-jxz%2BAAw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAHU4sP8-t%3Duqie%3Dz5pjb2ab7Te51q%2BWAac7pECeJrM%3DyrnDT7w%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAHU4sP8-t%3Duqie%3Dz5pjb2ab7Te51q%2BWAac7pECeJrM%3DyrnDT7w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/pNgeukzPL3A/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEM624Z4xuDmxXsL-fFP3rB4X%3D5_dwZpoeTZ6kfFg6VkE0db7A%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAEM624Z4xuDmxXsL-fFP3rB4X%3D5_dwZpoeTZ6kfFg6VkE0db7A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHU4sP-eCbbWu%2B01EuTwmaYwZBwN_REPUS0QpwdfpMraY57Y0w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
