Thanks Ted, replied inline. On Mon, Jul 18, 2011 at 8:52 AM, Ted Dunning <[email protected]> wrote: > To amplify slightly, there are a range of possible strategies you can use in > the disconnect scenario. > > One, is to have a special "master in waiting" state that is entered as soon > as a disconnect is seen. This is the most common strategy. > ------- I think my main worry is that the "is seen" can not be done by application thread fast enough, because the ping/event detection is part of the Sender/Event Thread, which can be scheduled arbitrarily later than the main thread.
> A second option is to continue operating as master for a some part of the > session expiration period. This is appropriate when not having a master is > a really bad thing and you want to minimize the time that this can happen. > Of course, if you can't reach ZK, the chances that you can function as a > master are somewhat limited. > > A variant on the second option would be to serve as a read-only master > during the time period between the disconnect event and the estimated > session expiration. > > The second and third options imply that you trust your clock which may be a > bad assumption in a virtual environment like EC2. > > On Mon, Jul 18, 2011 at 6:51 AM, Fournier, Camille F. < > [email protected]> wrote: > >> If the zk cluster doesn't get pings from your existing master, the zk >> client on that master should see a disconnected state event, not a node >> deletion event. Upon seeing that event, it should stop acting as master >> until such time as it can determine whether it has reconnected and is still >> master, or it reconnects and sees that its original session has failed or >> the master node is deleted. >> >> C >> >> >> >> ----- Original Message ----- >> From: Yang <[email protected]> >> To: [email protected] <[email protected]> >> Sent: Mon Jul 18 04:00:04 2011 >> Subject: Re: help on Zookeeper code walk through? >> >> Thanks Camille and Ben. >> >> I get the basic picture now. >> >> I have another question: in a leader election scenario (for example >> HBase Master election), I want to make sure that at any time , there >> is only at most one node running as master, and there is indeed one >> running as master all the time except for very short failover time >> period. >> >> then if only the connection between current master and ZK is down, >> ZK senses the lack of pings, and kills the session and ephemeral child >> node owned by the leader, and the next client node kicks in as leader. >> at this time, if the current leader machine is still working fine, its >> traffic going out to the its application servers as normal, would it >> be blissfully still acting as a leader, and violate our "single >> master" goal? for example if the Watcher.process() catches the >> nodeDelete event, and tries to set some var to stop the application >> server, but if this thread is stopped before the var is set, and is >> never invoked again, then the application server could just keep >> happily going along...? >> >> for example, the following dummy code >> >> class MyApplication { >> volatile boolean should_stop = false; >> class MyZKWatcher implements Zookeeper.Watcher { >> public void process(Event e) { >> if ( e is nodeDelete of my owner node ) { >> should_stop = true ; //************* >> } >> } >> >> public void runApp() { >> zk = new ZooKeeper(hostPort, 3000, this); >> while ( ! should_shop ) { >> send_out_some_messages to my application servers >> assuming I'm leader >> } >> } >> >> public static void main(String args[]) { >> new MyApplication().runApp(); >> } >> } >> >> >> basically if the nodeDelete event is caught but the Watcher stops >> right at "//*****" line , then the >> application main loop could still be going on?? otherwise I have to >> put a node exists() check before I send out every application message? >> >> >> Thanks a lot >> Yang >> 7 PM, Benjamin Reed <[email protected]> wrote: >> > if you are running with multiple servers, it is the leader that >> > declares sessions dead, so the leader will call killSession(). the >> > followers track the liveness of the clients with pings and will >> > periodically send liveness summaries to the leader. >> > >> > see camille's email the specific classes to look at. >> > >> > ben >> > >> > On Sat, Jul 16, 2011 at 1:44 AM, Yang <[email protected]> wrote: >> >> I'm wondering if a client loses session to its ephemeral znode, under >> >> the hood, how >> >> is the watcher triggered? >> >> >> >> went through the code , and found something that looks related: >> >> >> ZKDataBase.killSession()-->DataTree.killSession()--->DataTree.deleteNode()--->WatchManager.triggerWatch()--->Watcher.process() >> >> >> >> but how is ZKDataBase.killSession() called? from the info given in >> >> >> http://zookeeper.apache.org/doc/r3.3.3/zookeeperProgrammers.html#ch_zkSessions >> >> I can see the ZooKeeper client code does periodically ping the server >> >> to maintain liveness. but how the server checks for this liveness and >> >> trigger killSession(), here I'm having difficulty connecting the dots. >> >> >> >> could you please give me some help walking through this piece of code? >> >> >> >> Thanks >> >> Yang >> >> >> > >> >
