On Thu, Oct 21, 2010 at 9:08 AM, Sean Bigdatafun
> Can a lost znode be recovered automatically? Say, in a 3 znodes Zookeeper
> cluster, the cluster get into a critical status if a znode is lost. If I
> bring that lost znode back into running, can it rejoin the quorum?
Note that you have a terminological confusion here.
A znode is a Zookeeper data object roughly equivalent to a file. You don't
mean this, I think.
A Zookeeper server or node is a computer that is running a copy of
Zookeeper. You mean this.
The answer is, yes, the node can rejoin. Moreover, a new node can join and
you can progressively update the configuration
for the existing nodes one at a time.
> If it can, then that means my zookeeper cluster can run forever if I can
> somehow take care of my znodes (say, running a watchdog);
Correct. You can even keep your cluster running across minor upgrades by
using the rolling restart
idea from above. Likewise, if you decide to patch and reboot the machines
involved, you can do it
one machine at a time and avoid any downtime for the ZK cluster itself.
> if it can not, then that means my zookeeper cluster will need get restarted
> after a long
> period of time (because you will lose the total zookeeper cluster after
> losing two znodes -- and that can happen if there is not mechanism for
> to rejoin)
This is not a worry.
It is common for ZK clusters to last > 1 year. Because of the rolling
restart trick, the uptime for
the ZK cluster can easily exceed the uptime for single machine.
In some ways, a ZK cluster is a bit like the story of the Abe Lincoln's ax.
(See http://en.wikipedia.org/wiki/Ship_of_Theseus or