Henry Robinson commented on ZOOKEEPER-107:

I agree with pretty much everything I've read here, (in particular, the 
importance of getting consensus!), but wanted to clarify my initial comment a 

Rather than choose between strategies 1 and 2 as outlined by Benjamin, I think 
there's a hybrid approach needed. 

If a node is a member of a quorate cluster, then the most up to date membership 
information should be available to it in a znode. I think this is the most 
elegant approach, and is trivially achieved by pushing join/leave requests 
through the atomic broadcast pipeline.

If a node is joining the cluster, it needs to be able to bootstrap the location 
of the cluster from somewhere. There therefore needs to be a externally 
available resource containing a list of machines in the cluster that is at 
least accurate for one machine (as a joining node will try all servers in that 
list in turn). When I say available at some URI, this is what I mean. 
Currently, this information is kept statically at a URI that addresses 
conf/zoo.cfg on the local filesystem. I suggest generalising that to a general 
URI. One nice property is that it then does not tie a cluster to a particular 
machine, as the URI provides a level of indirection.

It is then the cluster administrator's responsibility to keep this URI 
up-to-date (although of course this should be automated), possibly via a client 
that just pulls membership information from the cluster periodically. As I said 
earlier, it's only important for the contents of this list to have one node in 
common with the true membership of the cluster, so it's allowed to get a bit 
out of sync. We can certainly easily imagine ways that ZK can help here. Of 
course the URI must be highly available, but it also has to exist, otherwise we 
could have 'orphaned' clusters that are running on machines whose identity we 
don't necessarily know. The URI can be a front for almost any scheme we like - 
periodic heartbeating of live nodes is one. 

The format of this file can be anything at all - from a serialised snapshot to 
a list of ip:port pairs, as long as it contains enough information for a client 
to find the cluster. Personally I would prefer human readable, simple formats.

To talk about recovery for a moment: when a node recovers from a crash and 
rejoins the cluster, it can help the cluster elect a master if the cluster is 
current non-quorate. This is because it was originally part of the cluster, and 
therefore the protocol guarantees that a quorum of nodes including the 
recovering one will have seen all committed proposals (this is important to 

If the node was not originally a member of the cluster, it must not help get a 
master elected as it cannot be part of a quorum. Similarly, a node cannot query 
the cluster to find out if it was originally a member because the quorum 
required to do so might not exist. Therefore every node that ever successfully 
joins a cluster must store this fact in its own persistent storage, as only it 
can know whether it is permitted to help run the election. 

Finally, the startup problem. Given a URI, nodes can bootstrap themselves onto 
a cluster simply by being told to start in startup mode. Alternatively, a 
single node can be distinguished (again, in the URI contents perhaps) which 
will start in single-node mode and process join requests one-by-one. 

> Allow dynamic changes to server cluster membership
> --------------------------------------------------
>                 Key: ZOOKEEPER-107
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-107
>             Project: Zookeeper
>          Issue Type: Improvement
>          Components: server
>            Reporter: Patrick Hunt
> Currently cluster membership is statically defined, adding/removing hosts 
> to/from the server cluster dynamically needs to be supported.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to