I think the ZK transaction logs can diverge from one another in some corner
cases. I have one such corner case listed below, could you please confirm if my
understanding is correct?
Imagine a 5 srever ensemble (A,B,C,D,E). All the servers are @ zxid 9. A is the
leader and it starts a new PROPOSAL (@zxid 10). A writes the proposal to the
log, so A moves to zxid 10. Others haven't received the PROPOSAL yet and A
crashes. Now the following happens:
1. B is elected as the newleader. B bumps up its in-mem zxid to 10. Since other
nodes are at the same zxid, it sends a SNAP so that the others can rebuild
their data tree. In-memory zxid of all other nodes moves to 10.
2. A comes back now, it accepts B as the leader as soon as the leader (B) and
N/2 other nodes vouch for B as the leader. So A joins the ensemble. Every
zookeeper node is at zxid 10.
3. A new request is submitted to B. B runs PROPOSAL and COMMIT phases and the
cluster moves up to zxid 11. But the transaction log of A is different from
that of everyone else now. So the transaction logs have diverged.
Could you confirm if this can happen? Or am I reading the code wrong?