> 1. When the 4th ZK node joins the cluster, does it take on the observer
> role since a quorum cannot be reached with the new node? Can I still
> connect my clients to it and create/remove nodes and receive events?
No, it joins as a normal member unless you've configured it as an
observer. Note that with 4 nodes you now need 3 running to get a
majority, which is why even numbers aren't recommended.
> 2. In the event 1 of the 3 voting nodes fails, will this 4th node become
> a voting member of the ensemble?
If configured as an observer it remains an observer.
> 3. When a new node comes online, it may have a different ip than the
> previous node. Do I need to update all node configurations and perform
> a rolling restart, or will simply connecting the new node to the
> existing ensemble make all nodes aware it is running?
Unfortunately ZK doesn't have any kind of dynamic configuration like
that currently. You need to update all the config files and restart