Faster detection of dead ephemeral nodes upon ZooKeeper restart?

[GIRARD-REYDET Vincent]

Hi ZooKeeper developers,

I'm trying to increase the speed at which ZooKeeper detects dead (expired) 
ephemeral nodes. I'm working with the release 3.4.6. My clients use Curator 
2.4.1 to connect to ZooKeeper and create their ephemeral nodes.

Currently, it takes sessionTimeout milliseconds to detect that an ephemeral 
node has expired. I have applied the ZOOKEEPER-922 patch to fasten the 
detection when the client who created the ephemeral nodes has crashed. In fact, 
I have modified it as is:
In NIOServerCnxn.java:
private void touchAndClose() {
   if(zkServer != null)
       sessionTimeout = 0; // Force expiration at the next tick
   try {
      if(zkServer != null)
         zkServer.touch(this);
   } catch (MissingSessionException e) {
      LOG.trace("Session missing?", e);
   }
   close();
}

With this, I have covered the case when a client crashes. At the next tick, the 
NIOServerCnxn will fail to send a ping and the session will be closed on the 
tick after.
However, there's one case that is not covered:

1.       Kill the quorum (kill -9 for example)

2.       Kill the client that has created ephemeral nodes

3.       Restart the quorum
In that case, ZooKeeper will recreate the ephemeral nodes from the snapshot, 
but they will only expire after the sessionTimeout. What I'de like to do is 
have them expire as soon as possible. I was thinking to the following change:
In SessionTrackerImpl.java:

public SessionTrackerImpl(SessionExpirer expirer,
        ConcurrentHashMap<Long, Integer> sessionsWithTimeout, int tickTime,
        long sid)
{
    super("SessionTracker");
    this.expirer = expirer;
    this.expirationInterval = tickTime;
    this.sessionsWithTimeout = sessionsWithTimeout;
    nextExpirationTime = roundToInterval(System.currentTimeMillis());
    this.nextSessionId = initializeNextSession(sid);
    for (Entry<Long, Integer> e : sessionsWithTimeout.entrySet()) {
        // When re-adding sessions with timeouts, wait first at most
        // 2 tick times. The client will revalidate the session anyway
        addSession(e.getKey(), 2*tickTime);
    }
}

This would leave the client 2*tickTime to re-register with one of the servers. 
From what I could see, when a session is re-registered, there's a 
touchSession(), so the sessionTimeout would be adapted anyway to anything 
negotiated.

I fear however that this would leave the door open to a corner case where the 
client would fail to register back in time, and the ephemeral nodes would be 
deleted while the client is still there. Do you think this could happen? Not 
that I already see some occasions with short session timeouts (10s with a 2s 
tick time) where the client fails to register in time, or the session expires 
on client side, and the ephemeral nodes are re-created by Curator.

Do you see a danger here? Do you think there's a better way of doing? I have 
read the threads about ZOOKEEPER-922 but it doesn't seem there's a clear 
opinion out of it.

Thanks for your help,
Vincent

*******************************

This e-mail contains information for the intended recipient only. It may 
contain proprietary material or confidential information. If you are not the 
intended recipient you are not authorised to distribute, copy or use this 
e-mail or any attachment to it. Murex cannot guarantee that it is virus free 
and accepts no responsibility for any loss or damage arising from its use. If 
you have received this e-mail in error please notify immediately the sender and 
delete the original email received, any attachments and all copies from your 
system.