Hi Ivan, Does this mean that if a note comes back and a replication is underway we'll end up with a node holding 2 replicas and 1 node holding node ?
Scenario: Node A - Replica 2 Node B - Replica 3 Node C - Replica 1 If node A dies and Node B get's Replica 2, as soon as node A (or a replacement) is brought up, is the final configuration likely to be Node A (or replcament) - No replicas Node B .- Replica 3 and 2 Node C - Replica 1 or is there a re-balance that takes place ? Thanks, Gonçalo Gonçalo Luiz On 10 July 2014 22:11, Ivan Brusic <[email protected]> wrote: > It's only been around for 3.5 years: > https://github.com/elasticsearch/elasticsearch/issues/623 :) > > I should clarify part of my previous statement. > > *"By default, the ongoing recovery is not cancelled when the missing node > rejoins the cluster. You can change the gateway settings [2] to control > when recovery kicks in."* > > What I meant to say is that an ongoing recovery is never cancelled once it > has commenced, no matter what settings. By default, recovery happens > immediately, but can be changed with the gateway settings. > > -- > Ivan > > > On Thu, Jul 10, 2014 at 1:48 PM, [email protected] < > [email protected]> wrote: > >> Indeed, auto_expand_replicas "all" triggers an update cluster settings >> action each time a node is added. >> >> Still blown by the many settings Elasticsearch provides. Feeling small. >> Homework: collecting a gist textfile of all ES 1.2 settings. >> >> Jörg >> >> >> On Thu, Jul 10, 2014 at 9:57 PM, Ivan Brusic <[email protected]> wrote: >> >>> Sticking to your use case, you might want to use >>> the auto_expand_replicas setting to "all" [1]: Never used it, but it sounds >>> what you are looking for. >>> >>> By default, the ongoing recovery is not cancelled when the missing node >>> rejoins the cluster. You can change the gateway settings [2] to control >>> when recovery kicks in. >>> >>> [1] >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html >>> [2] >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html >>> >>> Cheers, >>> >>> Ivan >>> >>> >>> On Thu, Jul 10, 2014 at 12:39 PM, Gonçalo Luiz <[email protected]> >>> wrote: >>> >>>> I get it know. >>>> >>>> I agree that setting the number of replicas is connected to the >>>> deployment reality in each case and it's derived variables and thus there >>>> is no one formula to fit all cases (it would't be a setting in that case). >>>> >>>> What I was trying to cover was the theoretical / extreme case where any >>>> node may fail at any time and what is the best way to go to minimize the >>>> chance of losing data. Also, in the case you want to scale down the >>>> installation (pottentially down to one node) without having to worry about >>>> selecting nodes that hold different replicated shards is an example that >>>> can beneffit from such configuration. >>>> >>>> I'm however not clear yet on what happens when a node goes down >>>> (triggering extra replication amongst the survivors) and then comes up >>>> again. Is the ongoing replication cancelled and the returning node brought >>>> up to date? >>>> >>>> Thanks for your valuable input. >>>> >>>> G. >>>> On 10 Jul 2014 18:07, "[email protected]" <[email protected]> >>>> wrote: >>>> >>>>> All I say is that it depends on the probability of the event of three >>>>> nodes failing simultaneously, not on the total number of nodes having a >>>>> replica. You can even have 5 nodes and the probability of the event of 4 >>>>> nodes failing simultaneously, and so on. >>>>> >>>>> As an illustration, suppose you have a data center with two >>>>> independent electric circuits and the probability of failure corresponds >>>>> with power outage, then it is enough to distribute nodes equally over >>>>> servers using the two independent power lines in the racks. If one >>>>> electric >>>>> circuit (plus UPS) fails, half of the nodes go down. With replica level 1, >>>>> ES cluster will keep all the data. There is no need to set replica level >>>>> equal to node number. >>>>> >>>>> Jörg >>>>> >>>>> >>>>> On Thu, Jul 10, 2014 at 8:55 AM, Gonçalo Luiz <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Joe, >>>>>> >>>>>> Thanks for your reply. >>>>>> On this thougth: >>>>>> >>>>>> " >>>>>> From my view your idea of better fault tolerance does not make much >>>>>> sense. The replica number is a statistical entity that is related to the >>>>>> probability of faults. The higher the replica, the higher the probability >>>>>> of surviving faults. There is no correlation to the total number of nodes >>>>>> in a cluster to ensure better fault tolerance. The fault tolerance >>>>>> depends >>>>>> on the probability of a node failure." >>>>>> >>>>>> I'm not getting it. If we have 4 nodes with 2 replicas it means that >>>>>> 3 of the nodes will have data of a given index (assuming 0 shards to ease >>>>>> the discussion), ritght? If those three nodes fail simultaneously the 4th >>>>>> will have no way of grabbing a copy and data will be lost forever. >>>>>> However >>>>>> if nr of replicas is 3, the 4th would be able to keep serving the >>>>>> requesrs >>>>>> and eventually handover a copy to a new node joining the cluster. >>>>>> How does this not help fault tolerance? I'm I missing something? >>>>>> >>>>>> Thanks, >>>>>> G. >>>>>> On 10 Jul 2014 00:21, "[email protected]" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> 1. You can set replica number at index creation time or by cluster >>>>>>> update settings >>>>>>> action >>>>>>> org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsAction >>>>>>> >>>>>>> 2. You will get an index with lower replica number :) >>>>>>> >>>>>>> 3. Yes. Quick code example: >>>>>>> >>>>>>> ClusterState clusterState = clusterService.state(); >>>>>>> // find number of data nodes >>>>>>> int numberOfDataNodes = 0; >>>>>>> for (DiscoveryNode node : clusterState.getNodes()) { >>>>>>> if (node.isDataNode()) { >>>>>>> numberOfDataNodes++; >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> 4. Yes. Use org.elasticsearch.cluster.ClusterStateListener >>>>>>> >>>>>>> From my view your idea of better fault tolerance does not make much >>>>>>> sense. The replica number is a statistical entity that is related to the >>>>>>> probability of faults. The higher the replica, the higher the >>>>>>> probability >>>>>>> of surviving faults. There is no correlation to the total number of >>>>>>> nodes >>>>>>> in a cluster to ensure better fault tolerance. The fault tolerance >>>>>>> depends >>>>>>> on the probability of a node failure. >>>>>>> >>>>>>> From the viewpoint of balancing load, it makes much sense. When >>>>>>> setting replica number to the number of nodes, the cluster can balance >>>>>>> search requests to all nodes which is optimal. >>>>>>> >>>>>>> Jörg >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 9, 2014 at 11:57 PM, <[email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I'm considering using elasticsearch as a repository for a PoC I'm >>>>>>>> currently developing. >>>>>>>> >>>>>>>> This PoC models an application that needs durability but not >>>>>>>> isolability, so I'm fine with the eventual consistency of reads >>>>>>>> against the >>>>>>>> most recent writes. >>>>>>>> >>>>>>>> As durability is paramount (we can't affort to lose the data unless >>>>>>>> 100% of the nodes die) I've been exploring the option of setting every >>>>>>>> shard to have N replicas where N is the number of nodes in the cluster. >>>>>>>> >>>>>>>> From what I've read so far it is possible to dynamically set the >>>>>>>> number or replicas which triggers a replication throttled replication >>>>>>>> process. >>>>>>>> >>>>>>>> I would like to have some help on the following steps (I'm running >>>>>>>> ES in embedded mode in a Java application): >>>>>>>> >>>>>>>> 1 - How can I set the number or replicas using the native Java >>>>>>>> client ? >>>>>>>> 2 - What happens if a node dies and the number of replicas is >>>>>>>> lowered to the number of surviving ones? >>>>>>>> 3 - Is it possible, from a participating node, to access the list >>>>>>>> of nodes in the cluster so I can use their count to set the number of >>>>>>>> replicas (step 1) ? >>>>>>>> 4 - is it possible to hook a callback to the event of a node >>>>>>>> joining or leaving the cluster ? >>>>>>>> >>>>>>>> I envisioning the following mechanism: >>>>>>>> >>>>>>>> a) - Start with one node, a given number of shards and 1 replica >>>>>>>> b)- Each time a node joins I adjust the number or replicas to match >>>>>>>> the new node count. In this case, there would be 2 replicas >>>>>>>> c) - An arbitrary number of nodes might be added and I'd execute >>>>>>>> step b) accordingly >>>>>>>> d) - At any time a node might leave the cluster and thus I need to >>>>>>>> lower the number or replicas to the new node count (I assume that the >>>>>>>> cluster would go ahead and proceed to compensate the lost replica by >>>>>>>> asking >>>>>>>> an existing node to hold 2 replicas instead of one; is this stopped by >>>>>>>> lowering the number or replicas?) >>>>>>>> >>>>>>>> >>>>>>>> The ultimate goal is to make sure no data is loss unless 100% of >>>>>>>> the nodes die before a new one can acquire a full replica. >>>>>>>> >>>>>>>> Is this doable? Does this make sense at all ? >>>>>>>> >>>>>>>> For the time being, I'm not worried about lack of disk space or >>>>>>>> bandwidth as I'm still in the very early days of the PoC. >>>>>>>> >>>>>>>> Thank you very much for all your work and help. >>>>>>>> >>>>>>>> Gonçalo >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/276418fa-812f-4af5-94a0-7362f5ba7931%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/276418fa-812f-4af5-94a0-7362f5ba7931%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/elasticsearch/hPvVz20v6YY/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGuUAJQYkkRtTU1H90MKpRg46dM1T2PzcP2Mfk1X8mbfA%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGuUAJQYkkRtTU1H90MKpRg46dM1T2PzcP2Mfk1X8mbfA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/CAMrm09qtY%2B_xoC2dXVXVzaXFxV70Q_XGs%2BrXoWqnSu_SrVvGJw%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAMrm09qtY%2B_xoC2dXVXVzaXFxV70Q_XGs%2BrXoWqnSu_SrVvGJw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/hPvVz20v6YY/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv1Ab6ite%2BHfaVh9ti2ea69Bh0ASjkCM8vE6%2Bn7j3PgQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv1Ab6ite%2BHfaVh9ti2ea69Bh0ASjkCM8vE6%2Bn7j3PgQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAMrm09r7EU4ZMy4%2BfVZ3SXmTPbh4YDsJgB_S0LWt3tRJHNihWg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAMrm09r7EU4ZMy4%2BfVZ3SXmTPbh4YDsJgB_S0LWt3tRJHNihWg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBhiw5DqC_W3mwCBKBOBjt95tRwj31J9Zq5XP_ROLsSTg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBhiw5DqC_W3mwCBKBOBjt95tRwj31J9Zq5XP_ROLsSTg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcroqUqU7an5B8bHQd_GFns4QsGCML5eS5LT6yiEDf6Q%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcroqUqU7an5B8bHQd_GFns4QsGCML5eS5LT6yiEDf6Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/hPvVz20v6YY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA_LtOdS-6Ht_DR57P%2BXkLWp5%3DV5Dz%2BVh2_cMkgy6kDSw%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA_LtOdS-6Ht_DR57P%2BXkLWp5%3DV5Dz%2BVh2_cMkgy6kDSw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMrm09rNQGaaMLE0iOD3NbFWNo7FtmzLh-JswwcKr3Ggp0ztfg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
