Patrick: I logged https://issues.apache.org/jira/browse/ZOOKEEPER-1428
If you feel there is anything missing in the JIRA, feel free to add it. Thanks for your help on this issue. Cheers On Tue, Mar 20, 2012 at 9:42 AM, Patrick Hunt <[email protected]> wrote: > On Tue, Mar 20, 2012 at 9:32 AM, Ted Yu <[email protected]> wrote: > > Near term, if we can find out a way for shell script to detect the > absence > > of particular zookeeper node, rolling-restart.sh can be restored. > > Otherwise we may need to remove it. > > I just tested this out with 3.4, and I see the following for statting > a non-existant znode: > > [zk: (CONNECTED) 1] stat /foobar > Node does not exist: /foobar > > vs statting one that does exist: > > [zk: (CONNECTED) 2] stat / > cZxid = 0x0 > ctime = Wed Dec 31 16:00:00 PST 1969 > mZxid = 0x0 > mtime = Wed Dec 31 16:00:00 PST 1969 > pZxid = 0x0 > cversion = -1 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 0 > numChildren = 1 > > You can look for "^Node does not exist" in the stat output instead of > checking the exit code. This would get around the problem until a more > permanent solution could be found. > > I hear you re time bound (i'd love to work on this myself). In that > case, would you mind creating a jira based on my suggestion of having > a new command line tool, give your hbase case as an example and any > requirements you might think of. Perhaps Hartmut or one of the other > contributors might be interested to work on this. > https://issues.apache.org/jira/browse/ZOOKEEPER > > Patrick > > > > > On Tue, Mar 20, 2012 at 9:16 AM, Patrick Hunt <[email protected]> wrote: > > > >> On Tue, Mar 20, 2012 at 6:57 AM, Ted Yu <[email protected]> wrote: > >> > I looked at the patch for ZOOKEEPER-1059 which should have converted > the > >> > NPE to KeeperException.NoNodeException > >> > > >> > Why would 'zkcli stat' command return 0 in case hbase master znode > >> expires ? > >> > > >> > Advice is appreciated. > >> > >> Hi Ted, sorry to see you're having troubles. I think I see the > >> disconnect. ZooKeeperMain is first and foremost a user shell. As such > >> it should not exit unless the quit command is run (or killed > >> explicitly, etc...). In this case ZOOKEEPER-1059 is fixing a bug in > >> the shell. It indeed is converting the NPE into a NoNodeException, > >> which the shell then converts into an error message to the user, and > >> continues. Prior to this patch the shell was failing on the NPE, which > >> then generated the non-0 exit from the process. > >> > >> Note that trunk has some further improvements along these lines that > >> you might also run into at some point in the future (3.5+): > >> > >> https://issues.apache.org/jira/browse/ZOOKEEPER-271 > >> https://issues.apache.org/jira/browse/ZOOKEEPER-1391 > >> https://issues.apache.org/jira/browse/ZOOKEEPER-1307 > >> > >> I think what we need is to have a tool that's intended for use both > >> programmatically and by humans, with more strict requirements about > >> input, output formatting and command handling, etc... Please see the > >> work Hartmut has been doing as part of 271 on trunk (3.5.0). Perhaps > >> we can augment these new classes to also support such a tool. However > >> it should instead be a true command line tool, rather than an shell. > >> Would you be available to work on this? > >> > >> Patrick > >> > >> ps. bigtop is now helping to verify cross project compatibility, it > >> would be great if you could introduce some hbase tests that would > >> flag these breakages in future. When bigtop does it's integration (ie > >> runs the hbase tests using the corresponding version of zk) it would > >> find these problems. We'd catch it much earlier. Thanks! > >> > >> > >> > FYI Jon filed a JIRA for the issue below which is a blocker for HBase > >> trunk. > >> > > >> > On Tue, Mar 20, 2012 at 12:36 AM, Jonathan Hsieh <[email protected]> > >> wrote: > >> > > >> >> I'm trying to test HBASE-5589 -- to see if I can add an API call to > >> >> HMasterInterface and do a rolling-restart / upgrade on a live cluster > >> which > >> >> lead me down another rabbit hole. > >> >> > >> >> I'm wondering how rolling-restart.sh script worked in the past (I can > >> spend > >> >> more time setting up an older version to test this, but figured I'd > >> ask). > >> >> > >> >> I'm getting stuck when the bin/rolling-restart.sh tries to wait until > >> the > >> >> Master ZNode expires. In this particular case, the script seems to > hang > >> >> there forever (even after the /hbase/master ephemeral node expires). > >> >> > >> >> Here's the code in the script: > >> >> ---- > >> >> # make sure the master znode has been deleted before continuing > >> >> zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool > >> >> zookeeper.znode.parent` > >> >> if [ "$zparent" == "null" ]; then zparent="/hbase"; fi > >> >> zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool > >> >> zookeeper.znode.master` > >> >> if [ "$zmaster" == "null" ]; then zmaster="master"; fi > >> >> zmaster=$zparent/$zmaster > >> >> echo -n "Waiting for Master ZNode ${zmaster} to expire" > >> >> while bin/hbase zkcli stat $zmaster >/dev/null 2>&1; do > >> >> echo -n "." > >> >> sleep 1 > >> >> done > >> >> echo #force a newline > >> >> ---- > >> >> > >> >> The problem is that 'bin/hbase zkcli stat /hbase/master ...' seems to > >> >> always returns with $? == 0 regardless if the znode is present or not > >> >> present! I've checked with Patrick Hunt (ZK committer) and this the > >> >> expected behavior. The only non-zero retcodes are for abnormal exits > >> >> (exceptions thrown) > >> >> > >> >> Here's the ZK code I was looking through > >> >> > >> >> > >> > https://github.com/apache/zookeeper/blob/release-3.4.3/src/java/main/org/apache/zookeeper/ZooKeeperMain.java#L736 > >> >> > >> >> > >> >> > >> > https://github.com/apache/zookeeper/blob/release-3.4.3/src/java/main/org/apache/zookeeper/ZooKeeper.java#L980 > >> >> > >> >> > >> >> Thoughts? > >> >> > >> >> Jon. > >> >> > >> >> -- > >> >> // Jonathan Hsieh (shay) > >> >> // Software Engineer, Cloudera > >> >> // [email protected] > >> >> > >> >
