Thanks Ivan, I guess i didn't explain what I was trying to do.
Promoting the 2 observers to participant will be a manual step (as part of disaster recovery) to get the cluster up. During this manual step, if needed, we can shutdown/terminate the old AZ instances. We also have puppet managing configuration. Puppet module will be updated to reflect new cluster instances. So when, if the AZ comes up, puppet will see that these instances are no longer part of the zookeeper cluster and module will stop zookeeper service. Do you still think this will not work and run into split brain scenario? A side question: Will observers will always in sync with entire cluster? in other words when observers will be in sync with the quorum participants? Thanks S ________________________________ From: Ivan Kelly <iv...@apache.org> Sent: Tuesday, August 3, 2021 7:07 AM To: user@zookeeper.apache.org <user@zookeeper.apache.org> Subject: [EXTERNAL EMAIL] - Re: zookeeper ensemble on two AWS AZ > In case of AZ with participant goes down, can I dynamically, promote > observers to participants? What are the pitfalls of this approach? No, this won't work. There's no way to tell if an AZ is down or just unreachable. So if you have 3 in a unreachable AZ, and you "promote" 3 observers in the second AZ to participants, both can in theory make writes to the log and you'll end up with a split brain scenario. -Ivan
