[
https://issues.apache.org/jira/browse/IGNITE-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-28869:
-----------------------------------
Description:
Success Rate: 1.5% (Last 68 Runs)
67 failed
1 successful
[https://ci2.ignite.apache.org/test/5214046175177601731?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
*Root Cause Analysis*
_Failure Symptom_
The test IgniteWalRebalanceTest#testRebalanceCancelOnSupplyError fails on TC
with `CorruptedTreeException` wrapping `Invalid version for inner update`:
{code:java}
java.lang.AssertionError: Invalid version for inner update [name=cache,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], isNew=false,
entry=GridDhtCacheEntry [..., ver=GridCacheVersion [topVer=394892252,
order=1783412249558, nodeOrder=2, dataCenterId=0], ...]],
newVer=GridCacheVersion [topVer=394892251, order=1783412250062, nodeOrder=1,
dataCenterId=0]]
at
GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5385)
at
GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4738)
...
at
IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError(IgniteWalRebalanceTest.java:509)
{code}
_Mechanism_
GridCacheVersion comparison (GridCacheVersion.compareTo()) compares first by
topVer, then by order. The assertion `ATOMIC_VER_COMPARATOR.compare(entry.ver,
newVer) <= 0` fails because the new write's topVer (394892251) is less than the
recovered entry's topVer (394892252).
GridCacheVersion.topVer is computed as:
{code:java}
// GridCacheVersionManager.next():
return next(topVer + offset, ...);
// where:
// topVer = readyTopologyVersion().topologyVersion() (from partition
exchange)
// offset = (gridStartTime - TOP_VER_BASE_TIME) / 1000 (grows by 1 per
second) {code}
In the test, after stopAllGrids() + startGrid(0):
1. readyTopologyVersion() resets to a small value (~1–3) after partition
exchange completes
2. offset is recalculated from the new gridStartTime, which is only slightly
larger than the previous one
3. Recovered data retains `topVer = prevReadyTopVer(~3) + prevOffset` from the
previous cluster session
4. New writes receive `topVer = currReadyTopVer(~1) + newOffset`
On slow TC machines, `newOffset - prevOffset < 1` (insufficient time passed
between stop and start), so:
{code:java}
394892252 = 3 + prevOffset (recovered entry)
394892251 = 1 + newOffset (new write)
→ 394892251 < 394892252 → assertion failure{code}
On faster local machines, enough time passes between stopAllGrids() and
startGrid(0) for offset to grow by ≥2 seconds, compensating for the lower
readyTopologyVersion.
*Fix*
* *Keep the supplier (node 0) alive* — stop only nodes 1 and 2 — so {{topVer}}
stays monotonic and recovered entries never outrank new writes. This mirrors
{{testMultipleNodesFailHistoricalRebalance}} in the same file.
* *Add {{spi.waitForBlocked()}}* before injecting the WAL-read failure. A warm
supplier serves the historical rebalance fast enough that, without this
barrier, the rebalance can finish before the failure fires and the future
returns {{true}} (reproduced ~1/15 without it).
was:
Success Rate: 1.5% (Last 68 Runs)
67 failed
1 successful
[https://ci2.ignite.apache.org/test/5214046175177601731?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
*Root Cause Analysis*
_Failure Symptom_
The test IgniteWalRebalanceTest#testRebalanceCancelOnSupplyError fails on TC
with `CorruptedTreeException` wrapping `Invalid version for inner update`:
{code:java}
java.lang.AssertionError: Invalid version for inner update [name=cache,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], isNew=false,
entry=GridDhtCacheEntry [..., ver=GridCacheVersion [topVer=394892252,
order=1783412249558, nodeOrder=2, dataCenterId=0], ...]],
newVer=GridCacheVersion [topVer=394892251, order=1783412250062, nodeOrder=1,
dataCenterId=0]]
at
GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5385)
at
GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4738)
...
at
IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError(IgniteWalRebalanceTest.java:509)
{code}
_Mechanism_
GridCacheVersion comparison (GridCacheVersion.compareTo()) compares first by
topVer, then by order. The assertion `ATOMIC_VER_COMPARATOR.compare(entry.ver,
newVer) <= 0` fails because the new write's topVer (394892251) is less than the
recovered entry's topVer (394892252).
GridCacheVersion.topVer is computed as:
{code:java}
// GridCacheVersionManager.next():
return next(topVer + offset, ...);
// where:
// topVer = readyTopologyVersion().topologyVersion() (from partition
exchange)
// offset = (gridStartTime - TOP_VER_BASE_TIME) / 1000 (grows by 1 per
second) {code}
In the test, after stopAllGrids() + startGrid(0):
1. readyTopologyVersion() resets to a small value (~1–3) after partition
exchange completes
2. offset is recalculated from the new gridStartTime, which is only slightly
larger than the previous one
3. Recovered data retains `topVer = prevReadyTopVer(~3) + prevOffset` from the
previous cluster session
4. New writes receive `topVer = currReadyTopVer(~1) + newOffset`
On slow TC machines, `newOffset - prevOffset < 1` (insufficient time passed
between stop and start), so:
{code:java}
394892252 = 3 + prevOffset (recovered entry)
394892251 = 1 + newOffset (new write)
→ 394892251 < 394892252 → assertion failure{code}
On faster local machines, enough time passes between stopAllGrids() and
startGrid(0) for offset to grow by ≥2 seconds, compensating for the lower
readyTopologyVersion.
*Fix*
Add a 5-second pause between stopAllGrids() and startGrid(0) to ensure offset
grows by ≥5, which reliably compensates for the readyTopologyVersion drop
(typically 2–3):
Tradeoff: test duration increases by ~5 seconds (from ~10s to ~15s), but the
test becomes deterministic on slow CI.
> Fix IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError
> -----------------------------------------------------------
>
> Key: IGNITE-28869
> URL: https://issues.apache.org/jira/browse/IGNITE-28869
> Project: Ignite
> Issue Type: Test
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain, ise
> Attachments: _Apache_Ignite_2.x_Tests_PDS_Indexing_45239.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Success Rate: 1.5% (Last 68 Runs)
> 67 failed
> 1 successful
> [https://ci2.ignite.apache.org/test/5214046175177601731?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
> *Root Cause Analysis*
> _Failure Symptom_
> The test IgniteWalRebalanceTest#testRebalanceCancelOnSupplyError fails on TC
> with `CorruptedTreeException` wrapping `Invalid version for inner update`:
> {code:java}
> java.lang.AssertionError: Invalid version for inner update [name=cache,
> topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], isNew=false,
> entry=GridDhtCacheEntry [..., ver=GridCacheVersion [topVer=394892252,
> order=1783412249558, nodeOrder=2, dataCenterId=0], ...]],
> newVer=GridCacheVersion [topVer=394892251, order=1783412250062, nodeOrder=1,
> dataCenterId=0]]
> at
> GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5385)
> at
> GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4738)
> ...
> at
> IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError(IgniteWalRebalanceTest.java:509)
> {code}
> _Mechanism_
> GridCacheVersion comparison (GridCacheVersion.compareTo()) compares first by
> topVer, then by order. The assertion
> `ATOMIC_VER_COMPARATOR.compare(entry.ver, newVer) <= 0` fails because the new
> write's topVer (394892251) is less than the recovered entry's topVer
> (394892252).
> GridCacheVersion.topVer is computed as:
> {code:java}
> // GridCacheVersionManager.next():
> return next(topVer + offset, ...);
> // where:
> // topVer = readyTopologyVersion().topologyVersion() (from partition
> exchange)
> // offset = (gridStartTime - TOP_VER_BASE_TIME) / 1000 (grows by 1 per
> second) {code}
> In the test, after stopAllGrids() + startGrid(0):
> 1. readyTopologyVersion() resets to a small value (~1–3) after partition
> exchange completes
> 2. offset is recalculated from the new gridStartTime, which is only slightly
> larger than the previous one
> 3. Recovered data retains `topVer = prevReadyTopVer(~3) + prevOffset` from
> the previous cluster session
> 4. New writes receive `topVer = currReadyTopVer(~1) + newOffset`
> On slow TC machines, `newOffset - prevOffset < 1` (insufficient time passed
> between stop and start), so:
> {code:java}
> 394892252 = 3 + prevOffset (recovered entry)
> 394892251 = 1 + newOffset (new write)
> → 394892251 < 394892252 → assertion failure{code}
> On faster local machines, enough time passes between stopAllGrids() and
> startGrid(0) for offset to grow by ≥2 seconds, compensating for the lower
> readyTopologyVersion.
> *Fix*
> * *Keep the supplier (node 0) alive* — stop only nodes 1 and 2 — so
> {{topVer}} stays monotonic and recovered entries never outrank new writes.
> This mirrors {{testMultipleNodesFailHistoricalRebalance}} in the same file.
> * *Add {{spi.waitForBlocked()}}* before injecting the WAL-read failure. A
> warm supplier serves the historical rebalance fast enough that, without this
> barrier, the rebalance can finish before the failure fires and the future
> returns {{true}} (reproduced ~1/15 without it).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
