I've been thinking about the implications of running Zookeeper in a fully dynamic distributed system, in which the number of nodes can be as few as one, or can be quite large. This has led to a few questions.
The dynamic server reconfiguration work appears to require a working quorum of servers under the old config in order to distribute the new config. This implies that the mechanism cannot be used if a quorum is lost (a common-mode failure across many servers). This leads to the obvious question, how does one recover from a (semi-)permanent loss of quorum? This would seem to require the HOG (Hand Of God) approach, with an external agent restarting the ZK servers with a new (shorter) server list. Presumably, the loss of quorum means a potential loss of state, since updates may not have made it to any of the surviving servers. If servers come to the ensemble with disparate contents, how does ZK converge on the new state? From what I've been able to read, it appears that all servers will end up converging to the state of the newly elected leader (and so any divergent contents on other nodes are discarded). Is this the case? If the system is to be fully dynamic, we have to deal with the two-node problem. How best to do this? In a two-node ensemble, it is guaranteed that if one of the nodes fails, the other node is guaranteed to be consistent, true? So if there is an external mechanism to prevent split brain, it should be possible to restart the surviving node in standalone mode, and once the second node returns, restarting both nodes should still guarantee consistency, yes? Thanks in advance, --Dave
