Re: Managing multi-site clusters with Zookeeper
If you can stand the latency for updates then zk should work well for you. It is unlikely that you will be able to better than zk does and still maintain correctness. Do note that you can, probalbly bias client to use a local server. That should make things more efficient. Sent from my iPhone On Mar 7, 2010, at 3:00 PM, Mahadev Konar wrote: The inter-site links are a nuisance. We have two data-centres with 100Mb links which I hope would be good enough for most uses, but we need a 3rd site - and currently that only has 2Mb links to the other sites. This might be a problem.
Re: Managing multi-site clusters with Zookeeper
Martin, 2Mb link might certainly be a problem. We can refer to these nodes as ZooKeeper servers. Znodes is used to data elements in the ZooKeeper data tree. The Zookeeper ensemble has minimal traffic which is basically health checks between the members of the ensemble. We call one of the members as Leader who is leading the ensemble and the others as Followers. The Leader does periodic health checks to see if the Followers are doing fine. This is of the order of << 1KB/sec. There is some traffic when the leader election within the ensemble happens. This might be of the order of 1-2KB/sec. As you mentioned the reads happen locally. So, a good enough link within the ensemble members is important so that these followers can be up to date with the Leader. But again looking at your config, looks like its mostly read only traffic. One more thing you should be aware of: Lets says a ephemeral node was created and the client died, then the clients connected to the slow ZooKeeper server (with 2Mb/s links) would lag behind the other clients connected to the other servers. As per my opinion you should do some testing since 2Mb/sec seems a little dodgy. Thanks mahadev On 3/7/10 2:09 PM, "Martin Waite" wrote: > Hi Mahadev, > > The inter-site links are a nuisance. We have two data-centres with 100Mb > links which I hope would be good enough for most uses, but we need a 3rd > site - and currently that only has 2Mb links to the other sites. This might > be a problem. > > The ensemble would have a lot of read traffic from applications asking which > database to connect to for each transaction - which presumably would be > mostly handled by local zookeeper servers (do we call these "nodes" as > opposed to znodes ?). The write traffic would be mostly changes to > configuration (a rare event), and changes in the health of database servers > - also hopefully rare. I suppose the main concern is how much ambient > zookeeper system chatter will cross the links. Are there any measurements > of how much traffic is used by zookeeper in maintaining the ensemble ? > > Another question that occurs is whether I can link sites A,B, and C in a > ring - so that if any one site drops out, the remaining 2 continue to talk. > I suppose that if the zookeeper servers are all in direct contact with each > other, this issue does not exist. > > regards, > Martin > > On 7 March 2010 21:43, Mahadev Konar wrote: > >> Hi Martin, >> As Ted rightly mentions that ZooKeeper usually is run within a colo >> because >> of the low latency requirements of applications that it supports. >> >> Its definitely reasnoble to use it in a multi data center environments but >> you should realize the implications of it. The high latency/low throughput >> means that you should make minimal use of such a ZooKeeper ensemble. >> >> Also, there are things like the tick Time, the syncLimit and others (setup >> parameters for ZooKeeper in config) which you will need to tune a little to >> get ZooKeeper running without many hiccups in this environment. >> >> Thanks >> mahadev >> >> >> On 3/6/10 10:29 AM, "Ted Dunning" wrote: >> >>> What you describe is relatively reasonable, even though Zookeeper is not >>> normally distributed across multiple data centers with all members >> getting >>> full votes. If you account for the limited throughput that this will >> impose >>> on your applications that use ZK, then I think that this can work well. >>> Probably, you would have local ZK clusters for higher transaction rate >>> applications. >>> >>> You should also consider very carefully whether having multiple data >> centers >>> increases or decreases your overall reliability. Unless you design very >>> carefully, this will normally substantially degrade reliability. Making >>> sure that it increases reliability is a really big task that involves a >> lot >>> of surprising (it was to me) considerations and considerable hardware and >>> time investments. >>> >>> Good luck! >>> >>> On Sat, Mar 6, 2010 at 1:50 AM, Martin Waite >> wrote: >>> Is this a viable approach, or am I taking Zookeeper out of its >> application domain and just asking for trouble ? >>> >>> >> >>
Re: Managing multi-site clusters with Zookeeper
Hi Mahadev, The inter-site links are a nuisance. We have two data-centres with 100Mb links which I hope would be good enough for most uses, but we need a 3rd site - and currently that only has 2Mb links to the other sites. This might be a problem. The ensemble would have a lot of read traffic from applications asking which database to connect to for each transaction - which presumably would be mostly handled by local zookeeper servers (do we call these "nodes" as opposed to znodes ?). The write traffic would be mostly changes to configuration (a rare event), and changes in the health of database servers - also hopefully rare. I suppose the main concern is how much ambient zookeeper system chatter will cross the links. Are there any measurements of how much traffic is used by zookeeper in maintaining the ensemble ? Another question that occurs is whether I can link sites A,B, and C in a ring - so that if any one site drops out, the remaining 2 continue to talk. I suppose that if the zookeeper servers are all in direct contact with each other, this issue does not exist. regards, Martin On 7 March 2010 21:43, Mahadev Konar wrote: > Hi Martin, > As Ted rightly mentions that ZooKeeper usually is run within a colo > because > of the low latency requirements of applications that it supports. > > Its definitely reasnoble to use it in a multi data center environments but > you should realize the implications of it. The high latency/low throughput > means that you should make minimal use of such a ZooKeeper ensemble. > > Also, there are things like the tick Time, the syncLimit and others (setup > parameters for ZooKeeper in config) which you will need to tune a little to > get ZooKeeper running without many hiccups in this environment. > > Thanks > mahadev > > > On 3/6/10 10:29 AM, "Ted Dunning" wrote: > > > What you describe is relatively reasonable, even though Zookeeper is not > > normally distributed across multiple data centers with all members > getting > > full votes. If you account for the limited throughput that this will > impose > > on your applications that use ZK, then I think that this can work well. > > Probably, you would have local ZK clusters for higher transaction rate > > applications. > > > > You should also consider very carefully whether having multiple data > centers > > increases or decreases your overall reliability. Unless you design very > > carefully, this will normally substantially degrade reliability. Making > > sure that it increases reliability is a really big task that involves a > lot > > of surprising (it was to me) considerations and considerable hardware and > > time investments. > > > > Good luck! > > > > On Sat, Mar 6, 2010 at 1:50 AM, Martin Waite >wrote: > > > >> Is this a viable approach, or am I taking Zookeeper out of its > application > >> domain and just asking for trouble ? > >> > > > > > >
Re: Managing multi-site clusters with Zookeeper
Hi Martin, As Ted rightly mentions that ZooKeeper usually is run within a colo because of the low latency requirements of applications that it supports. Its definitely reasnoble to use it in a multi data center environments but you should realize the implications of it. The high latency/low throughput means that you should make minimal use of such a ZooKeeper ensemble. Also, there are things like the tick Time, the syncLimit and others (setup parameters for ZooKeeper in config) which you will need to tune a little to get ZooKeeper running without many hiccups in this environment. Thanks mahadev On 3/6/10 10:29 AM, "Ted Dunning" wrote: > What you describe is relatively reasonable, even though Zookeeper is not > normally distributed across multiple data centers with all members getting > full votes. If you account for the limited throughput that this will impose > on your applications that use ZK, then I think that this can work well. > Probably, you would have local ZK clusters for higher transaction rate > applications. > > You should also consider very carefully whether having multiple data centers > increases or decreases your overall reliability. Unless you design very > carefully, this will normally substantially degrade reliability. Making > sure that it increases reliability is a really big task that involves a lot > of surprising (it was to me) considerations and considerable hardware and > time investments. > > Good luck! > > On Sat, Mar 6, 2010 at 1:50 AM, Martin Waite wrote: > >> Is this a viable approach, or am I taking Zookeeper out of its application >> domain and just asking for trouble ? >> > >
