On Thu, Oct 21, 2010 at 9:08 AM, Sean Bigdatafun <sean.bigdata...@gmail.com>wrote:
> Can a lost znode be recovered automatically? Say, in a 3 znodes Zookeeper > cluster, the cluster get into a critical status if a znode is lost. If I > bring that lost znode back into running, can it rejoin the quorum? > Yes. Note that you have a terminological confusion here. A znode is a Zookeeper data object roughly equivalent to a file. You don't mean this, I think. A Zookeeper server or node is a computer that is running a copy of Zookeeper. You mean this. The answer is, yes, the node can rejoin. Moreover, a new node can join and you can progressively update the configuration for the existing nodes one at a time. > If it can, then that means my zookeeper cluster can run forever if I can > somehow take care of my znodes (say, running a watchdog); Correct. You can even keep your cluster running across minor upgrades by using the rolling restart idea from above. Likewise, if you decide to patch and reboot the machines involved, you can do it one machine at a time and avoid any downtime for the ZK cluster itself. > if it can not, then that means my zookeeper cluster will need get restarted > after a long > period of time (because you will lose the total zookeeper cluster after > losing two znodes -- and that can happen if there is not mechanism for > znode > to rejoin) > This is not a worry. It is common for ZK clusters to last > 1 year. Because of the rolling restart trick, the uptime for the ZK cluster can easily exceed the uptime for single machine. In some ways, a ZK cluster is a bit like the story of the Abe Lincoln's ax. (See http://en.wikipedia.org/wiki/Ship_of_Theseus or http://www.nytimes.com/books/first/m/mansfield-ax.html)
