Forgive me if this is documented somewhere, but I can't find an
answer. Our application is not enormous, so we will be using 4
application nodes that will also initially run Zookeeper. As our load
increases, Zookeeper will be moved to nodes that only run ZK and no
other processes. Given that we will initially only have 4 nodes in our
cluster and I have a few questions around the semantics of an even
number of nodes.
1. When the 4th ZK node joins the cluster, does it take on the observer
role since a quorum cannot be reached with the new node? Can I still
connect my clients to it and create/remove nodes and receive events?
2. In the event 1 of the 3 voting nodes fails, will this 4th node become
a voting member of the ensemble?
3. When a new node comes online, it may have a different ip than the
previous node. Do I need to update all node configurations and perform
a rolling restart, or will simply connecting the new node to the
existing ensemble make all nodes aware it is running?