The logs shows that a node was not contactable and so was removed from the cluster. That could be a load issue on the node, network issues, or something else. Check the logs on the node that disconnected and see what you find.
If you leave the defaults, you risk the chance of a split-brain situation, where nodes lose contact with each other and then decide to become a master. This can cause you to end up with multiple masters in multiple clusters, which is not good for data integrity. Setting the min masters means that if a node cannot contact N masters, it will refuse connections. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 14 January 2014 10:46, Mohit Anchlia <[email protected]> wrote: > I am trying to understand why I should change it? This setting looks like > the right setting in most cases. Did you see anything out of ordinary in > that log message I pasted? > > On Mon, Jan 13, 2014 at 3:40 PM, David Pilato <[email protected]> wrote: > >> No it's not the default setting. >> min master node = total num of nodes / 2 + 1 >> >> You have 5 nodes, 5 / 2 + 1 = 3 >> >> >> >> -- >> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* >> @dadoonet <https://twitter.com/dadoonet> | >> @elasticsearchfr<https://twitter.com/elasticsearchfr> >> >> >> Le 14 janvier 2014 at 00:06:34, Mohit Anchlia >> ([email protected]<//[email protected]>) >> a écrit: >> >> Is the default setting 3? How do I go about selecting right configuration >> setting for this property? >> >> On Mon, Jan 13, 2014 at 3:01 PM, David Pilato <[email protected]> wrote: >> >>> So I think you should first set minimum master node to 3. See >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery >>> >>> Probably you need to close your isolated node and restart it. >>> >>> >>> >>> -- >>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* >>> @dadoonet <https://twitter.com/dadoonet> | >>> @elasticsearchfr<https://twitter.com/elasticsearchfr> >>> >>> >>> Le 13 janvier 2014 at 23:58:33, Mohit Anchlia >>> ([email protected]<//[email protected]>) >>> a écrit: >>> >>> I have 5 nodes. I am using all the default settings. >>> >>> On Mon, Jan 13, 2014 at 2:32 PM, David Pilato <[email protected]> wrote: >>> >>>> It sounds like a split brain issue or something. >>>> >>>> How many nodes do you have? >>>> Did you set minimum master node? >>>> >>>> -- >>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* >>>> @dadoonet <https://twitter.com/dadoonet> | >>>> @elasticsearchfr<https://twitter.com/elasticsearchfr> >>>> >>>> >>>> Le 13 janvier 2014 at 22:48:32, Mohit Anchlia >>>> ([email protected]<//[email protected]>) >>>> a écrit: >>>> >>>> Thanks! It looks like one of the nodes even though it was up still >>>> it went out of cluster. Is there no retry? >>>> >>>> >>>> [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] >>>> master {new >>>> [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, >>>> previous >>>> [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, >>>> removed >>>> {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, >>>> reason: zen-disco-master_failed >>>> ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300 >>>> ]]{group=group2}) >>>> >>>> [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] >>>> removed >>>> {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, >>>> reason: >>>> zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), >>>> reason failed to ping, tried [3] times, each with maximum [30s] timeout >>>> >>>> >>>> On Mon, Jan 13, 2014 at 1:34 PM, David Pilato <[email protected]> wrote: >>>> >>>>> Yellow means that some replicas are not allocated. >>>>> If you have only one node and have index set with one replica >>>>> (default), yellow is the expected state. >>>>> >>>>> >>>>> -- >>>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* >>>>> @dadoonet <https://twitter.com/dadoonet> | >>>>> @elasticsearchfr<https://twitter.com/elasticsearchfr> >>>>> >>>>> >>>>> Le 13 janvier 2014 at 22:30:23, Mohit Anchlia >>>>> ([email protected]<//[email protected]>) >>>>> a écrit: >>>>> >>>>> Cluster state is showing yellow. What's the best way to find why >>>>> it's in yellow state? >>>>> >>>>> It has been up and stable so far except when I looked today it shows >>>>> state turned into yellow. >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com >>>>> . >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. >>>>> >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com. >>>> >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/etPan.52d46968.5577f8e1.376c%40MacBook-Air-de-David.local. >>>> >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrsU-k9u6hir2CWoobhnaOTD_3Bjk3Xf_ZjMDSBkB0DeQ%40mail.gmail.com. >>> >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/etPan.52d47036.77465f01.376c%40MacBook-Air-de-David.local. >>> >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWqbrUh5p2t74Bc_6kL%3DxjAcLGrJzJ6wt_fRQqy6Dj-DKg%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/etPan.52d47982.5e884adc.376c%40MacBook-Air-de-David.local >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAOT3TWqx_KLF_p8nJ4m3On-nSvRncY9OdHa8KS1HDjcRHhfQhQ%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aGEw8HqnMP--Fb-U%2BCjB58VBFb1bQenk_RmenNKAmMHg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
