there are two things to keep in mind when thinking about this issue:

1) if a zk client is disconnected from the cluster, the client is essentially in limbo. because the client cannot talk to a server it cannot know if its session is still alive. it also cannot close its session.

2) the client only finds out about session expiration events when the client reconnects to the cluster. if zk tells a client that its session is expired, the ephemerals that correspond to that session will already be cleaned up.

one of the main design points about zk is that zk only gives correct information. if zk cannot give correct information, it basically says "i don't know". connection loss exceptions and disconnected states are basically "i don't know".

generally applications we design go into a "safe" mode, meaning they may serve reads but reject changes, when disconnected from zk and only kill themselves when they find out their session has expired.


ps - session information is replicated to all zk servers, so if a leader dies, all replicas know the sessions that are currently active and their timeouts.

On 08/16/2010 09:03 PM, Ted Dunning wrote:
Ben or somebody else will have to repeat some of the detailed logic for
this, but it has
to do with the fact that you can't be sure what has happened during the
network partition.
One possibility is the one you describe, but another is that the partition
happened because
a majority of the ZK cluster lost power and you can't see the remaining
nodes.  Those nodes
will continue to serve any files in a read-only fashion.  If the partition
involves you losing
contact with the entire cluster at the same time a partition of the cluster
into a quorum and
a minority happens, then your ephemeral files could continue to exist at
least until the breach
in the cluster itself is healed.

Suffice it to say that there are only a few strategies that leave you with a
coherent picture
of the universe.  Importantly, you shouldn't assume that the ephemerals will
disappear at
the same time as the session expiration event is delivered.

On Mon, Aug 16, 2010 at 8:31 PM, Qing Yan<>  wrote:

Ouch, is this the current ZK behavior? This is unexpected, if the
client get partitioned from ZK cluster, he should
get notified and take some action(e.g. commit suicide) otherwise how
to tell a ephemeral node is really
up or down? Zombie can create synchronization nightmares..

On Mon, Aug 16, 2010 at 7:22 PM, Dave Wright<>  wrote:
Another possible cause for this that I ran into recently with the c
client -
you don't get the session expired notification until you are reconnected
the quorum and it informs you the session is lost.  If you get
and can't reconnect you won't get the notification.  Personally I think
client api should track the session expiration time locally and
you once it's expired.

On Aug 16, 2010 2:09 AM, "Qing Yan"<>  wrote:

Hi Ted,

  Do you mean GC problem can prevent delivery of SESSION EXPIRE event? you have met this problem before?
I didn't see any OOM though, will look into it more.

On Mon, Aug 16, 2010 at 12:46 PM, Ted Dunning<>
I am assuming that y...

Reply via email to