Re: ZK recovery questions
I did try a quick test on Windows (yes, some of us use Windows :) I thought simply changing the dataDir to the /dev/null equivalent on Windows would do the trick. It didn't work. It looks like a Java issue because I noticed inconsistencies in the File API regarding this. I wrote about it here - http://javaforu.blogspot.com/2010/07/devnull-on-windows.html devnull-on-windows . BTW the Windows equivalent is nul. This is the error I got on Windows (below). The mkdirs() returns false. As noted on my blog, it returns true for some cases. 2010-07-20 22:25:47,851 - FATAL [main:zookeeperserverm...@62] - Unexpected exception, exiting abnormally java.io.IOException: Unable to create data directory nul:\version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.init(FileTxnSnapLog.java:79) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:102) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:85) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:51) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:108) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) Ashwin. -- View this message in context: http://zookeeper-user.578899.n2.nabble.com/ZK-recovery-questions-tp5310116p5319775.html Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: getChildren() when the number of children is very large
On Tue, Jul 20, 2010 at 8:47 PM, André Oriani aori...@gmail.com wrote: Ted, just to clarify. By file you mean znode, right ? Yes. So you are advising me to try an atomic append to znode's by first calling getData and then trying to conditionally set the data by using the version information obtained in the previous step ? Exactly.
Re: ZK recovery questions
i did a benchmark a while back to see the effect of turning off the disk. (it wasn't as big as you would think.) i had to modify the code. there is an option to turn off the sync in the config that will get you most of the performance you would get by turning off the disk entirely. ben On 07/20/2010 11:01 PM, Ashwin Jayaprakash wrote: I did try a quick test on Windows (yes, some of us use Windows :) I thought simply changing the dataDir to the /dev/null equivalent on Windows would do the trick. It didn't work. It looks like a Java issue because I noticed inconsistencies in the File API regarding this. I wrote about it here - http://javaforu.blogspot.com/2010/07/devnull-on-windows.html devnull-on-windows . BTW the Windows equivalent is nul. This is the error I got on Windows (below). The mkdirs() returns false. As noted on my blog, it returns true for some cases. 2010-07-20 22:25:47,851 - FATAL [main:zookeeperserverm...@62] - Unexpected exception, exiting abnormally java.io.IOException: Unable to create data directory nul:\version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.init(FileTxnSnapLog.java:79) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:102) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:85) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:51) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:108) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) Ashwin.
Re: ZK recovery questions
My own experiments in my own environment where ZK is being used purely for coordination at a fairly low transaction rate (tens to hundreds of ops per second, mostly status updates) made me feel that disk throughput would only be detectable as an issue for pretty massively abused ZK applications. The impact of disk writing is surprisingly small even for pretty high throughput cases and for moderate or low throughput, it is just not detectable. Those seem to share a lot with the applications that could benefit from being able to restart new servers efficiently from disk snapshot and log and having the ability to restart the entire cluster with previous state. On Wed, Jul 21, 2010 at 9:28 AM, Benjamin Reed br...@yahoo-inc.com wrote: i did a benchmark a while back to see the effect of turning off the disk. (it wasn't as big as you would think.) i had to modify the code. there is an option to turn off the sync in the config that will get you most of the performance you would get by turning off the disk entirely. ben On 07/20/2010 11:01 PM, Ashwin Jayaprakash wrote: I did try a quick test on Windows (yes, some of us use Windows :) I thought simply changing the dataDir to the /dev/null equivalent on Windows would do the trick. It didn't work. It looks like a Java issue because I noticed inconsistencies in the File API regarding this. I wrote about it here - http://javaforu.blogspot.com/2010/07/devnull-on-windows.html devnull-on-windows . BTW the Windows equivalent is nul. This is the error I got on Windows (below). The mkdirs() returns false. As noted on my blog, it returns true for some cases. 2010-07-20 22:25:47,851 - FATAL [main:zookeeperserverm...@62] - Unexpected exception, exiting abnormally java.io.IOException: Unable to create data directory nul:\version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.init(FileTxnSnapLog.java:79) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:102) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:85) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:51) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:108) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) Ashwin.
Adding observers
Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
Sorry as an addendum I had one more question. (1) If I snapshot the data on other observer machines will I be able to bootstrap new observers with it? Given that writes are like a one time thing. Cheers A On Wed, Jul 21, 2010 at 10:30 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
On Wed, Jul 21, 2010 at 10:30 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: (1) Is it possible to increase the number of observers in the cluster dynamically? Not quite, but practically speaking you can do as good as this. In general, pretty much any ZK configuration change can be done without service interruption by using a rolling restart. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Others will give more authoritative answers, but I am pretty sure that the limitation on the number of observers is strictly related to write rate x number of observers. This is related to the fact that writes need to come from the current master. It isn't hard to imagine how to write a reflector that watches for all changes and writes these to a secondary cluster. That would essentially eliminate the limit on number of observers. Something like that may already be possible within the current system (I couldn't say since I haven't looked into observers that much).
Re: Adding observers
Hi Avinash - (1) Is it possible to increase the number of observers in the cluster dynamically? Not really - you can use the same rolling-restart technique that is often mentioned on this list for adding servers to the ensemble, but you can't add them and expect that ZK will auto-find them. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? This will be possible, but there is some overhead in communicating with observers as well as normal voting followers. However, given that writes are rare, perhaps this kind of overhead would be acceptable? Henry Any insight would be very helpful. Thanks A -- Henry Robinson Software Engineer Cloudera 415-994-6679
Re: Adding observers
Any example scripts for the rolling restart technique that anyone would be kind enough to share? Thanks Avinash On Wed, Jul 21, 2010 at 10:44 AM, Henry Robinson he...@cloudera.com wrote: Hi Avinash - (1) Is it possible to increase the number of observers in the cluster dynamically? Not really - you can use the same rolling-restart technique that is often mentioned on this list for adding servers to the ensemble, but you can't add them and expect that ZK will auto-find them. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? This will be possible, but there is some overhead in communicating with observers as well as normal voting followers. However, given that writes are rare, perhaps this kind of overhead would be acceptable? Henry Any insight would be very helpful. Thanks A -- Henry Robinson Software Engineer Cloudera 415-994-6679
Re: Adding observers
It is really simpler than you can imagine. Something like this should be plenty sufficient. for h in ZK_HOSTS do ssh $h $ZK_HOME/bin/zkServer.sh restart sleep 5 done This is just something I typed in, not something I checked. It is intended to give you the idea. I will leave it to you to fix my silly errors. :-) Note that you probably don't need to do this to the observers since they don't need to know about other observers. On Wed, Jul 21, 2010 at 10:48 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Any example scripts for the rolling restart technique that anyone would be kind enough to share?
Re: Do implementations of Watcher need to be thread-safe?
as long as a watcher object is only used with a single ZooKeeper object it will be called by the same thread. ben On 07/21/2010 11:12 AM, Joshua Ball wrote: Hi, Do implementations of Watcher need to be thread-safe, or can I assume that process(...) will always be called by the same thread? Thanks, Josh Ua Ball
Re: Do implementations of Watcher need to be thread-safe?
Thanks! On Wed, Jul 21, 2010 at 11:23 AM, Benjamin Reed br...@yahoo-inc.com wrote: as long as a watcher object is only used with a single ZooKeeper object it will be called by the same thread. ben On 07/21/2010 11:12 AM, Joshua Ball wrote: Hi, Do implementations of Watcher need to be thread-safe, or can I assume that process(...) will always be called by the same thread? Thanks, Josh Ua Ball
