Adding observers
Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
Sorry as an addendum I had one more question. (1) If I snapshot the data on other observer machines will I be able to bootstrap new observers with it? Given that writes are like a one time thing. Cheers A On Wed, Jul 21, 2010 at 10:30 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
On Wed, Jul 21, 2010 at 10:30 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: (1) Is it possible to increase the number of observers in the cluster dynamically? Not quite, but practically speaking you can do as good as this. In general, pretty much any ZK configuration change can be done without service interruption by using a rolling restart. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Others will give more authoritative answers, but I am pretty sure that the limitation on the number of observers is strictly related to write rate x number of observers. This is related to the fact that writes need to come from the current master. It isn't hard to imagine how to write a reflector that watches for all changes and writes these to a secondary cluster. That would essentially eliminate the limit on number of observers. Something like that may already be possible within the current system (I couldn't say since I haven't looked into observers that much).
Re: Adding observers
Hi Avinash - (1) Is it possible to increase the number of observers in the cluster dynamically? Not really - you can use the same rolling-restart technique that is often mentioned on this list for adding servers to the ensemble, but you can't add them and expect that ZK will auto-find them. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? This will be possible, but there is some overhead in communicating with observers as well as normal voting followers. However, given that writes are rare, perhaps this kind of overhead would be acceptable? Henry Any insight would be very helpful. Thanks A -- Henry Robinson Software Engineer Cloudera 415-994-6679
Re: Adding observers
Any example scripts for the rolling restart technique that anyone would be kind enough to share? Thanks Avinash On Wed, Jul 21, 2010 at 10:44 AM, Henry Robinson he...@cloudera.com wrote: Hi Avinash - (1) Is it possible to increase the number of observers in the cluster dynamically? Not really - you can use the same rolling-restart technique that is often mentioned on this list for adding servers to the ensemble, but you can't add them and expect that ZK will auto-find them. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? This will be possible, but there is some overhead in communicating with observers as well as normal voting followers. However, given that writes are rare, perhaps this kind of overhead would be acceptable? Henry Any insight would be very helpful. Thanks A -- Henry Robinson Software Engineer Cloudera 415-994-6679
Re: Adding observers
It is really simpler than you can imagine. Something like this should be plenty sufficient. for h in ZK_HOSTS do ssh $h $ZK_HOME/bin/zkServer.sh restart sleep 5 done This is just something I typed in, not something I checked. It is intended to give you the idea. I will leave it to you to fix my silly errors. :-) Note that you probably don't need to do this to the observers since they don't need to know about other observers. On Wed, Jul 21, 2010 at 10:48 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Any example scripts for the rolling restart technique that anyone would be kind enough to share?
