sorry to jump in late.

if i understand the scenario correctly, you are partitioned from ZK, but you still have access to the NN on which you are holding leases to files. the problem is that even though your ephemeral nodes may timeout, you are still holding a lease on the NN and recovery would go faster if you actually closed the file. right? or is it deeper than that? can you open a file in such a way that you stomp the lease? or make sure that the lease timeout is smaller than the session timeout and only renew if you are still connected to ZK?


Jean-Daniel Cryans wrote:
If the machine was completely partitioned, as far as I know, it would lose
it's lease so the only thing we have to make sure about is clearing the
state of the region server by doing a "restart" so that it's ready to come
back in the cluster. If ZK is down but the rest is up, closing the files in
HDFS should ensure that we lose a minimum of data if not losing any.

I think that in a multi-rack setup it is possible to not be able to talk to
ZK but to be able to talk to the Namenode as machines can be anywhere.
Especially in HBase 0.20, the master can failover on any node that has a
backup Master ready. So in that case, the region server should consider
itself gone from the cluster and close any connection it has and restart.

Those are very legetimate questions Gustavo, thanks for asking.


On Wed, Jun 24, 2009 at 3:38 PM, Gustavo Niemeyer <>wrote:

Ben's opinion is that it should not belong in the default API but in the
common client that another recent thread was about. My opinion is just
I need such a functionality, wherever it is.
Understood, sorry.  I just meant that it feels like something that
would likely be useful to other people too, so might have a role in
the default API to ensure it gets done properly considering the
details that Ben brought up.

If the node gets the exception (or has it's own timer), as I wrote, it
shut itself down to release HDFS leases as fast as possible. If ZK is
down and it's not a network partition, then HBase is down and this is
because it won't be able to work anyway.
Right, that's mostly what I was wondering.  I was pondering about
under which circumstances the node would be unable to talk to the
ZooKeeper server but would still be holding the HDFS lease in a way
that prevented the rest of the system from going on.  If I understand
what you mean, if ZooKeeper is down entirely, HBase would be down for
good. If the machine was partitioned off entirely, the HDFS side of
things will also be disconnected, so shutting the node down won't help
the rest of the system recovering.

Gustavo Niemeyer

Reply via email to