Perhaps what you want is more of a messaging solution... Have you looked into, say ActiveMQ or RabbitMQ? You can use ZK for this sort of confirmation stuff but it feels a bit like overkill.
C -----Original Message----- From: Aleksey Yakovlev [mailto:[email protected]] Sent: Friday, August 26, 2011 6:06 PM To: [email protected] Subject: Re: Zookeeper on two clusters? Ted: So, if I find a way to talk to one cluster from another using let's say the netcat - it'll be enough for ZK, right? As for the architecture and motivation - there is a lot of people, who designed them and are managing them, and I'm not one of them. In my opinion these clusters are too loosely connected, so it's very difficult to build a distributed software system (which I'm doing) on them. A simple example: if one cluster sends a file to another, and this another one gets this file corrupted and needs to ask the first one to resend - it doesn't have a simple way to do that. Actually, it can send a short file as a confirmation - but this transaction can also fail. Then the first one will need to say "I'm waiting for confirmation - where is it?" So, we are talking about some communication protocol - which is already in the ZK, right? That's why I'm thinking about the ZK. The second cluster could send the confirmation to the ZK, and the first one could receive it from the ZK. And it'd be the ZK internal matter to replicate the message between its servers. BTW, what is the "observer" you mentioned? Thanks AY On Fri, Aug 26, 2011 at 5:04 PM, Ted Dunning <[email protected]> wrote: > Not so much mutual pingability, but mutual tcp-ability. > > But seriously, why not run ZK in one cluster and put an observer in the > other cluster? Or do that bi-directionally? > > Or put two machines in each cluster and a tie-breaker somewhere outside > like > Amazon's EC2? > > You haven't provided any real motivation for your original architecture. > What are you really trying to do? For instance, you say a "distributed > systems works in both centers" and needs "fail-over". What do you really > mean? > > Do you want both data centers to continue to operate in the even of > partition? In that case you need two separate ZK clusters, possibly with > some cross-datacenter ephemeral files. > > Do you want one data center to stop work if it loses contact with the > other? > In that case, put a single ZK cluster in one data-center. > > Do you want to ignore most partition issues, but make sure that exactly one > data-center is considered master at all times and that at least one is > considered master as much as possible? In that case, use the external > tie-breaker architecture. > > > On Fri, Aug 26, 2011 at 1:39 PM, Aleksey Yakovlev < > [email protected]> wrote: > > > Camille: > > > > Sorry to say, I can't ping one cluster from another... > > You are saying, that all the servers mutual pingability is necessary and > > sufficient for ZK to operate, right? > > > > Thanks > > AY > > > > On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. < > > [email protected]> wrote: > > > > > As long as the two clusters can ping each other, just set up a single > ZK > > > cluster spread across the two, 3 or 5 nodes (total, not per cluster). > > Note > > > that if these two clusters are used for business continuity purposes > > (spread > > > across 2 data centers), you still risk the outage of the zookeeper if > you > > > lose one of the data centers (the one containing the majority of the ZK > > > nodes). > > > > > > There's no reason to set up 2 ZK quorums and merge them. > > > > > > C > > > > > > -----Original Message----- > > > From: Aleksey Yakovlev [mailto:[email protected]] > > > Sent: Friday, August 26, 2011 10:09 AM > > > To: [email protected] > > > Subject: Re: Zookeeper on two clusters? > > > > > > Mahadev: > > > > > > I need these two clusters to communicate somehow, because our > distributed > > > system works on both of them - so we need some coordination, for > example, > > > to > > > recover after failures. I could create my own servers/protocols for > that, > > > but why - the ZooKeeper does just that, right? > > > > > > The number 5 isn't important here, configuration and communication is > > > > > > Thanks > > > AY > > > > > > On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar < > [email protected] > > > >wrote: > > > > > > > Aleksey, > > > > I am a little confused. Whats the motivation behind merging the > > > > quorums? Why do you want 5 on each cluster? > > > > > > > > thanks > > > > mahadev > > > > > > > > On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev > > > > <[email protected]> wrote: > > > > > Hi, > > > > > > > > > > We have two clusters but I can't just ssh from one cluster to > another > > - > > > I > > > > > have to enter a password, use some authentication, go through a > load > > > > > balancer etc > > > > > It would be very useful for us to use the Zookeeper as a common > > storage > > > > > place for small configuration data, accessible from both these > > > clusters. > > > > > > > > > > So, I'm thinking about starting like 5 servers on each cluster and > > > merge > > > > > them into a single quorum - anybody did that? > > > > > If yes, how to configure this double-side quorum? > > > > > > > > > > (I'm completely new to the Zookeeper) > > > > > > > > > > Thanks > > > > > AY > > > > > > > > > > > > > > >
