[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347026#comment-16347026 ]
Robert Joseph Evans commented on ZOOKEEPER-2845: ------------------------------------------------ I have a fix that I will be posting shortly. I need to clean up the patch and make sure that I get pull requests ready for all of the branches that ZOOKEEPER-2926 went into. The following table describes the situation that allows a node to get into an inconsistent state. || ||N1||N2||N3|| |Start with cluster in sync N1 is leader|0x0 0x5|0x0 0x5|0x0 0x5| |N2 and N3 go down|0x0 0x5| | | |Proposal to N1 (fails with no quorum)|0x0 0x6| | | |N2 and N3 return, but N1 is restarting. N2 elected leader| |0x1 0x0|0x1 0x0| |A proposal is accepted| |0x1 0x1|0x1 0x1| |N1 returns and is trying to sync with the new leader N2|0x0 0x6|0x1 0x1|0x1 0x1| At this point the code in {{LearnerHandler.syncFollower}} takes over to bring N1 into sync with N2 the new leader. That code checks the following in order # Is there a {{forceSync}}? Not in this case # Are the two zxids in sync already? No {{0x0 0x6 != 0x1 0x1}} # is the peer zxid > the local zxid (and peer didn't just rotate to a new epoch)? No {{0x0 0x6 < 0x1 0x1}} # is the peer zxid in between the max committed log and the min committed log? In this case yes it is, but it shouldn't be. The max committed log is {{0x1 0x1}}. The min committed log is {{0x0 0x5}} or something likely below it because it is based off of distance in the edit log. The issue is that once the epoch changes, {{0x0}} to {{0x1}}, the leader has no idea if the edits are in its edit log without explicitly checking for them. The reason that ZOOKEEPER-2926 exposed this is because previously when a leader was elected the in memory DB was dropped and everything was reread from disk. When this happens the {{0x0 0x6}} proposal was lost. But it is not guaranteed to be lost in all cases. In theory a snapshot could be taken triggered by that proposal, either on the leader, or on a follower that also allied the proposal, but does not join the new quorum in time. As such ZOOKEEPER-2926 really just extended the window of an already existing race. But it extended it almost indefinitely so it is much more likely to happen. My fix is to update {{LearnerHandler.syncFollower}} to only send a {{DIFF}} if the epochs are the same. If they are not the same we don't know if something we inserted that we don't know about. > Data inconsistency issue due to retain database in leader election > ------------------------------------------------------------------ > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum > Affects Versions: 3.4.10, 3.5.3, 3.6.0 > Reporter: Fangmin Lv > Assignee: Robert Joseph Evans > Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)