Hi, I've been reading the docs and trying out some basic Zookeeper examples.
I have a few simple questions related to recovery.
It would be good to have questions like these on the Wiki/docs to avoid
noobs like me asking the same thing over and over.
- If 1 out of 3 servers crashes and the log files are unrecoverable, how
do we provision a replacement server?
- If the server log is recoverable but provisioning takes a long time,
then what happens if the old log file is far behind the current state? The
docs say that recovery is based on fuzzy check pointing and snapshots but I
wasn't clear as to how long "catching up" would take
- What happens at the client side code if a server quorum is lost? Does
the ZK service freeze or continue to service just reads?
- If there was a temporary glitch (n/w or GC) and the replica to which
the client is connected breaks away from the quorum does the client get
notified? Does it stop processing client requests? Does it rejoin the
cluster without manual intervention?
- Now if even the client cannot connect to other servers (split brain)
.. ... well I suppose this question is moot
- Do the servers really have to run with file based persistence? I saw
that someone wanted this in-memory mode for unit testing (ZK
but there are cases where only a transient ZK service is needed. Most
enterprise systems have replicated Databases anyway. So, the fear of data
loss is minimal. If ZK logs are the only means of recovery, then this might
be harder to implement
- A client example with full fledged error handling would be very useful
for starters. I'm not sure if http://github.com/sgroschupf/zkclient and
http://code.google.com/p/cages/ have everything but they do look
promising. Plain ZK API is a bit overwhelming :)