Hi JZ, Sorry for being dense here, but my point is that: Suppose the leader latch does NOT setLeadership(false) on receiving SUSPENDED, then what you mentioned below won't happen, right?
*>>> Further, if zk2/zk3 are communicating then they are in Quorum. Therefore, latch2 will declare that it is leader* this way we can avoid the unnecessary leader switch due to the transient latch1 <--> zk1 connection loss (i.e., the reboot of zk1 somehome) if the latch1 can re-connect to the zk ensemble in time. On Mon, Mar 10, 2014 at 11:28 PM, Jordan Zimmerman < [email protected]> wrote: > In step 3, during the time that zk1 is partitioned latch1 cannot know if > it's leader or not. Further, if zk2/zk3 are communicating then they are in > Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MUST > go into a non-leader state. It doesn't matter if latch1 reconnects before > the session expires. The issue is how it manages its state while it is in > partition. To your point, latch1 could check to see if it's still leader > when it reconnects but I'm not sure that there's a huge win here. Any > clients of the latch will have to have handled the leadership loss during > the partition. > > -JZ > > > From: chao chu [email protected] > Reply: [email protected] [email protected] > Date: March 10, 2014 at 10:21:29 AM > To: user [email protected] > > Subject: Re: Leader Latch recovery after suspended state > > Thanks for your reply first. > > >>> Technically, this is already the case. When a RECONNECTED is > received, LeaderLatch will attempt to regain leadership. The problem is > that when there is a network partition there is no way to guarantee that > you are still the leader. If there is Quorum in another segment of the > cluster a new leader might be elected there. > > I don't quite understand what you meant here. But let me explain the > case I mentioned in detail: > > 1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two > participants for the leader electoin {leader latch1, leader latch2} > 2. latch1 is the current leader, and connected to zk1 > 3. the connection latch1 <--> zk1 broken somehow > 4. but sooner (within the session timeout), latch1 re-connected (maybe to > zk2) > > I guess the problem is that in LeaderLatch, it setLeadership(false) > (meaning that there must be leader changes) as long as a SUSPEND state (or > actually a ZK DISCONNECTED event). > > while in this case, ideally (just my personal thinking), since latch1 > re-connected in time, its znode will still be there, no others (latch2 in > this example) will observe any events due to this, and once re-connected, > if it detected that its znode still there, checkLeadership should still set > it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be > called) during this whole process. Thus, we can avoid the unnecessary > leader switch (which, as I mentioned, can be very expensive in most of > cases). > > does this make any sense to you? thanks > > > > On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman < > [email protected]> wrote: > >> Please provide an implementation/fix and submit a pull request on >> Github. >> >> I also have a related question about not only re-use the znode, but >> imho, It would be great that LeaderLatch can survive from teomprary >> ConnectionLossException (i.e., due to transient network issue). >> >> Technically, this is already the case. When a RECONNECTED is received, >> LeaderLatch will attempt to regain leadership. The problem is that when >> there is a network partition there is no way to guarantee that you are >> still the leader. If there is Quorum in another segment of the cluster a >> new leader might be elected there. >> >> -JZ >> >> From: chao chu [email protected] >> Reply: [email protected] [email protected] >> Date: March 10, 2014 at 9:39:50 AM >> To: [email protected] [email protected] >> Subject: Re: Leader Latch recovery after suspended state >> >> Hi, >> >> Just want to see if there is any progress on this? >> >> I also have a related question about not only re-use the znode, but imho, >> It would be great that LeaderLatch can survive from teomprary >> ConnectionLossException (i.e., due to transient network issue). >> >> I guess in most cases, the context switch due to leader re-election is >> quite expensive, we might not want to do that just because of some >> transient issue. if the current leader can re-connect within the session >> timeout, it should still hold the leadership and no leader change would >> happen during between. The similar rational like the differences between >> ConnestionLossException (which is recoverable) and SessionExipredException >> (which is not recoverable). >> >> what are your thoughts on this? Thanks a lot! >> >> Regards, >> >> >> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman < >> [email protected]> wrote: >> >>> Yes, I was suggesting how to patch Curator. >>> >>> On Aug 20, 2013, at 10:59 AM, Calvin Jia <[email protected]> wrote: >>> >>> Currently this is not supported in the Curator library, but the >>> Curator library (specifically leader latch's reset method) is the >>> correct/logical place to add this feature if I want it? >>> >>> >>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman < >>> [email protected]> wrote: >>> >>>> On reset() it could check to see if its node still exists. It would >>>> make the code a lot more complicated though. >>>> >>>> -JZ >>>> >>>> On Aug 20, 2013, at 10:25 AM, Calvin Jia <[email protected]> wrote: >>>> >>>> A leader latch enters the suspended state after failing to receive a >>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the >>>> timeout). For the last 1 third, it tries to contact another ZK machine. If >>>> it is successful, it will enter the state reconnected. >>>> >>>> However, on reconnect, despite the fact the original node it created in >>>> ZK is still there, it will create another ephemeral-sequential node (the >>>> reset method is called). This means it will relinquish leadership, if there >>>> is another machine with a latch in the same path. >>>> >>>> Is there any way to reconnect and reuse the original ZK node? >>>> >>>> Thanks! >>>> >>>> >>>> >>> >>> >> >> >> -- >> ChuChao >> >> > > > -- > ChuChao > > -- ChuChao
