Yonik Seeley wrote:
We have solr nodes create ephemeral znodes (name based on host and port).
The ephemeral znode takes some time to remove of course, so what
happens is that if I bounce a solr server (containing a zk client) the
ephemeral node will still exist when the server comes back up.  Since
it exists, the ephemeral won't be re-created, but it does disappear

Yes. This is expected behavior - and one of the main reasons for having low session timeouts. Unfortunately the alternative of having lower timeout can also have negative effect (sensitivity to network/resource issues, in particular GC as you know)

What's the best way to handle this situation?  Delete and re-create?
Watch it and re-create when it does disappear?
There's no way to "hand over" responsibility for an ephemeral znode, right?

There is one way, not really "hand over" but rather "take back".

What I mean is, when you establish a session you get a session id AND a password. The session is valid until either it is closed or expires. If, when your client (solr server) comes back up, it is able to provide the session id and password to the ZK service it will _recover_ the session. This probably means that you would have to write the sessionid/password to disk (if your client is crashing/killed for example) and handle some corner cases (like session expired). It's not a common case as you have to store the id persistently, but it is possible and might address your issue (assuming the primary issue is that your client goes down and becomes active again quickly).

A second alternative is to actively delete the znode in question. However this has similar issues to the prior paragraph - the client that's coming up would have to know the session id is had previously, so that it could get a Stat on the znode, compare the "ephemeralOwner" and delete if it was the previous owner.

However short of knowing the session id I don't see how you get around this issue other than just waiting for the timeout. The problem is how would you know to delete the ephemeral node in question? How is any other solr client going to know that the client died, and therefore remove the znode... Really that's what you are relying on zk for, via the timeout.

Yes, you could set a watch on the znode in question, when it is deleted (via the eventual session timeout) you would recreate.

Another option is that the client that's being restarted, it could write it's IP address somewhere (as part of the znode or part of the data say, or another znode that names ref to this znode, etc...). If the client was bounced it could compare it's IP to the ip "assigned" to the znode, if it was the same you would have confidence you were the owner (this one depends on the usecase but based on what you described it should work unless the client changes IPs dynamically).


Reply via email to