Re: Question regarding Membership Election
Hi Vijay, It might not have been clear in my previous message, but I was suggesting that with three locations you can tolerate one DC going down, assuming you have votes across locations. We have performed some preliminary tests some time ago with ZooKeeper servers across different locations, and it looked good. Perhaps we should think about running a more extensive set of tests and making them available for reference. Just in case you're interested, here is perhaps an interesting example of how you can tolerate one location going down with three locations. For each location, create a group of three ZooKeeper servers. In this setting, we need only two votes from a remote location, independent of where the leader is. For example, consider the following groups and servers in each group: Group 1: A B C Group 2: D E F Group 3: G H I If the leader is A, then it needs to collect votes only from two servers in group 1, say A and B, and two votes from some other location, say D and E in group 2 (corresponding to location 2). To make it work, all you have to do is create groups in ZooKeeper, which enables the hierarchical quorum feature. This example is interesting because it enables the system to mask one server crashing in each group/location and a whole location going down, and requires one fewer vote compared to absolute majority, which would be five servers given that we have nine total. Hope it is useful. -Flavio On Jan 14, 2010, at 11:00 PM, Vijay wrote: Hi Falvio, Yes i am concerned about the latency between the DC's (Across continents), We actually have 6 locations but how exactly are we going to do it if we have the 3rd DC? Regards, On Thu, Jan 14, 2010 at 1:46 PM, Flavio Junqueira inc.com> wrote: Hi Vijay, I'm just curious: why exactly you want all voting nodes in a single data center? Are you concerned about latency? It might not be possible in your case, but if you have a third location available, you would be able to tolerate one location going down. -Flavio On Jan 14, 2010, at 9:12 PM, Vijay wrote: Hi, I read about observers in other datacenter, My question is i dont want voting across the datacenters (So i will use observers), at the same time when a DC goes down i dont want to loose the cluster, whats the solution for it? I have to have 3 nodes in primary DC to accept 1 node failure. Thats fine... but what about the other DC? how many nodes and how will i make it work? Regards,
Re: Question regarding Membership Election
Hi Falvio, Yes i am concerned about the latency between the DC's (Across continents), We actually have 6 locations but how exactly are we going to do it if we have the 3rd DC? Regards, On Thu, Jan 14, 2010 at 1:46 PM, Flavio Junqueira wrote: > Hi Vijay, I'm just curious: why exactly you want all voting nodes in a > single data center? Are you concerned about latency? > > It might not be possible in your case, but if you have a third location > available, you would be able to tolerate one location going down. > > -Flavio > > > On Jan 14, 2010, at 9:12 PM, Vijay wrote: > > Hi, >> >> I read about observers in other datacenter, >> >> My question is i dont want voting across the datacenters (So i will use >> observers), at the same time when a DC goes down i dont want to loose the >> cluster, whats the solution for it? >> >> I have to have 3 nodes in primary DC to accept 1 node failure. Thats >> fine... >> but what about the other DC? how many nodes and how will i make it work? >> >> Regards, >> >> > >
Re: Question regarding Membership Election
Hi Vijay, I'm just curious: why exactly you want all voting nodes in a single data center? Are you concerned about latency? It might not be possible in your case, but if you have a third location available, you would be able to tolerate one location going down. -Flavio On Jan 14, 2010, at 9:12 PM, Vijay wrote: Hi, I read about observers in other datacenter, My question is i dont want voting across the datacenters (So i will use observers), at the same time when a DC goes down i dont want to loose the cluster, whats the solution for it? I have to have 3 nodes in primary DC to accept 1 node failure. Thats fine... but what about the other DC? how many nodes and how will i make it work? Regards,
Re: Question regarding Membership Election
Hi Vijay, Sadly there isnt any. It would be great to have someone contribute one to zookeeper code base :). Thanks mahadev On 1/14/10 12:58 PM, "Vijay" wrote: > Thanks Mahadev that helps, > > Is there any hookup's (In zoo keeper) or examples which i can take a look > for the bridging process? > > Regards, > > > > > > On Thu, Jan 14, 2010 at 12:38 PM, Mahadev Konar wrote: > >> Hi Vijay, >> Unfortunately you wont be able to keep running the observer in the other >> DC if the quorum in the DC 1 is dead. Most of the folks we have talked to >> also want to avoid voiting across colos. They usually run two instances of >> Zookeeper in 2 DC's and copy state of zookeeper (using a bridge) across >> colos to keep them in sync. Usually the data requirement across colos is >> very small and they are usually able to do that by copying data across with >> there own bridge process. >> >> Hope that helps. >> >> Thanks >> mahadev >> >> >> On 1/14/10 12:12 PM, "Vijay" wrote: >> >>> Hi, >>> >>> I read about observers in other datacenter, >>> >>> My question is i dont want voting across the datacenters (So i will use >>> observers), at the same time when a DC goes down i dont want to loose the >>> cluster, whats the solution for it? >>> >>> I have to have 3 nodes in primary DC to accept 1 node failure. Thats >> fine... >>> but what about the other DC? how many nodes and how will i make it work? >>> >>> Regards, >>> >> >>
Re: Question regarding Membership Election
Thanks Henry, Regards, On Thu, Jan 14, 2010 at 12:40 PM, Henry Robinson wrote: > Hi - > > If you put all your voting nodes in one datacenter, that datacenter becomes > a 'single point of failure' for the cluster. If it gets cut off from any > other datacenters, the cluster will not be available to those datacenters. > > If you want to withstand the failure of datacenters, then you need voting > members inside every datacenter. Observers can't suddenly become voting > members. > > You can't even put 'dormant' voting members (that never bother to vote) in > your other datacenters because you would need a quorum of them to continue > after the original datacenter failed. And if this was true, the original > datacenter would not, by construction, contain a quorum of voting nodes. So > you'd still have to vote outside the cluster. > > Henry > > 2010/1/14 Vijay > > > Hi, > > > > I read about observers in other datacenter, > > > > My question is i dont want voting across the datacenters (So i will use > > observers), at the same time when a DC goes down i dont want to loose the > > cluster, whats the solution for it? > > > > I have to have 3 nodes in primary DC to accept 1 node failure. Thats > > fine... > > but what about the other DC? how many nodes and how will i make it work? > > > > Regards, > > > > >
Re: Question regarding Membership Election
Thanks Mahadev that helps, Is there any hookup's (In zoo keeper) or examples which i can take a look for the bridging process? Regards, On Thu, Jan 14, 2010 at 12:38 PM, Mahadev Konar wrote: > Hi Vijay, > Unfortunately you wont be able to keep running the observer in the other > DC if the quorum in the DC 1 is dead. Most of the folks we have talked to > also want to avoid voiting across colos. They usually run two instances of > Zookeeper in 2 DC's and copy state of zookeeper (using a bridge) across > colos to keep them in sync. Usually the data requirement across colos is > very small and they are usually able to do that by copying data across with > there own bridge process. > > Hope that helps. > > Thanks > mahadev > > > On 1/14/10 12:12 PM, "Vijay" wrote: > > > Hi, > > > > I read about observers in other datacenter, > > > > My question is i dont want voting across the datacenters (So i will use > > observers), at the same time when a DC goes down i dont want to loose the > > cluster, whats the solution for it? > > > > I have to have 3 nodes in primary DC to accept 1 node failure. Thats > fine... > > but what about the other DC? how many nodes and how will i make it work? > > > > Regards, > > > >
Re: Question regarding Membership Election
Hi - If you put all your voting nodes in one datacenter, that datacenter becomes a 'single point of failure' for the cluster. If it gets cut off from any other datacenters, the cluster will not be available to those datacenters. If you want to withstand the failure of datacenters, then you need voting members inside every datacenter. Observers can't suddenly become voting members. You can't even put 'dormant' voting members (that never bother to vote) in your other datacenters because you would need a quorum of them to continue after the original datacenter failed. And if this was true, the original datacenter would not, by construction, contain a quorum of voting nodes. So you'd still have to vote outside the cluster. Henry 2010/1/14 Vijay > Hi, > > I read about observers in other datacenter, > > My question is i dont want voting across the datacenters (So i will use > observers), at the same time when a DC goes down i dont want to loose the > cluster, whats the solution for it? > > I have to have 3 nodes in primary DC to accept 1 node failure. Thats > fine... > but what about the other DC? how many nodes and how will i make it work? > > Regards, > >
Re: Question regarding Membership Election
Hi Vijay, Unfortunately you wont be able to keep running the observer in the other DC if the quorum in the DC 1 is dead. Most of the folks we have talked to also want to avoid voiting across colos. They usually run two instances of Zookeeper in 2 DC's and copy state of zookeeper (using a bridge) across colos to keep them in sync. Usually the data requirement across colos is very small and they are usually able to do that by copying data across with there own bridge process. Hope that helps. Thanks mahadev On 1/14/10 12:12 PM, "Vijay" wrote: > Hi, > > I read about observers in other datacenter, > > My question is i dont want voting across the datacenters (So i will use > observers), at the same time when a DC goes down i dont want to loose the > cluster, whats the solution for it? > > I have to have 3 nodes in primary DC to accept 1 node failure. Thats fine... > but what about the other DC? how many nodes and how will i make it work? > > Regards, >
