Dmitry Werner created IGNITE-29014:
--------------------------------------
Summary: Non-monotonic cache versions after fast full cluster
restart cause "Invalid version for inner update" and B+ tree corruption
Key: IGNITE-29014
URL: https://issues.apache.org/jira/browse/IGNITE-29014
Project: Ignite
Issue Type: Task
Reporter: Dmitry Werner
*Problem*
When a full cluster restart (all nodes stopped, then restarted) happens within
a few seconds, cache versions generated by the new cluster incarnation can be
LOWER than the versions of entries restored from the previous incarnation's
persistent storage. In a real environment this manifests as "Invalid version
for inner update" errors and B+ tree corruption (CorruptedTreeException) in the
logs after a fast grid restart.
*Root cause*
GridCacheVersionManager#onLocalJoin derives the cache version offset from the
cluster grid start time with 1-second precision:
offset = (gridStartTime - TOP_VER_BASE_TIME) / 1000
The leading field of GridCacheVersion is (topologyVersion + offset). After a
restart the topology version resets to a small value (1..N), so the offset has
to compensate for that. If the new incarnation starts within a few seconds of
the previous one, its offset is too small: versions generated at the new low
topology versions end up below versions persisted by the previous incarnation
(which reached a higher topology version while running). The first write to
such an entry then fails the version check in GridCacheMapEntry ("Invalid
version for inner update") and can lead to B+ tree corruption.
*Related work*
See IGNITE-28886. IgniteWalRebalanceTest.testWithLocalWalChange /
testWithGlobalWalChange reproduced exactly this scenario in a test environment:
starting a 2-node cluster right after stopping a 4-node (respectively 3-node)
cluster failed with "Invalid version for inner update" / B+ tree corruption.
The fix in IGNITE-28886 is a test-side workaround only — the test now waits at
least TOP_VER_OFFSET_WAIT_SECS (5) seconds between cluster incarnations. This
is a timeout stub that masks the underlying product issue: in a real
environment (planned maintenance,
full-cluster upgrade, failover) a fast restart can still corrupt data, and the
test's 5-second wait is not a product-level guarantee.
*Suggested fix (for discussion)*
Guarantee cache version monotonicity across incarnations: persist the maximum
generated cache version (e.g. the last data version) in durable storage, and on
startup initialize the version offset/order strictly above the persisted value
instead of relying solely on the wall-clock grid start time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
