Fangmin Lv created ZOOKEEPER-3114:
-------------------------------------
Summary: Real time consistency check between leader and learner
Key: ZOOKEEPER-3114
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3114
Project: ZooKeeper
Issue Type: New Feature
Components: quorum
Reporter: Fangmin Lv
Assignee: Fangmin Lv
Fix For: 3.6.0
The correctness of ZooKeeper was kind of proved in theory in ZAB paper, but the
implementation is a bit different from the paper, for example, save the
currentEpoch and proposals/commits upon to NEWLEADER is not atomic in current
implementation, so the correctness of ZooKeeper is not actually proved in
reality.
Also bugs could be introduced during implementation, issues like sending
NEWLEADER packet too early reported in ZOOKEEPER-3104 might be there since the
beginning (didn't check exactly when this was introduced).
More correctness issues were introduced when adding new features, like on disk
txn sync, local session, retain database, etc, both of these features added
inconsistency bugs on production.
To catch the consistency issue earlier, internally, we're running external
consistency checkers to compare nodes (digest), but that's not efficient (slow
and expensive) and there are corner cases we cannot cover in external checker.
For example, we don't know the last zxid before epoch change, which makes it's
impossible to check it's missing txn or not. Another challenge is the false
negative which is hard to avoid due to fuzzy snapshot or expected txn gap
during snapshot syncing, etc.
This Jira is going to propose a built-in real time consistency check by
calculating the digest of DataTree after applying each txn, and sending it over
to learner during propose time so that it can verify the correctness in real
time.
The consistency check will cover all phases, including loading time during
startup, syncing, and broadcasting. It can help us avoid data lost or data
corrupt due to bad disk and catch bugs in code.
The protocol change will make backward compatible to make sure we can
enable/disable this feature transparently.
As for performance impact, based on our testing, it will add a bit overhead
during runtime, but doesn't have obvious impact in general.
h2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
