[
https://issues.apache.org/jira/browse/IGNITE-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094496#comment-18094496
]
Ignite TC Bot commented on IGNITE-28869:
----------------------------------------
{panel:title=Branch: [pull/13323/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/13323/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *--> Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=9189568&buildTypeId=IgniteTests24Java8_RunAll]
{color:#ffffff}tcbot-analysis-comment chainBuildId=9189568
rerunBuildIds=none{color}
> Fix IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError
> -----------------------------------------------------------
>
> Key: IGNITE-28869
> URL: https://issues.apache.org/jira/browse/IGNITE-28869
> Project: Ignite
> Issue Type: Test
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain, ise
> Attachments: _Apache_Ignite_2.x_Tests_PDS_Indexing_45239.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Success Rate: 1.5% (Last 68 Runs)
> 67 failed
> 1 successful
> [https://ci2.ignite.apache.org/test/5214046175177601731?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
> *Root Cause Analysis*
> _Failure Symptom_
> The test IgniteWalRebalanceTest#testRebalanceCancelOnSupplyError fails on TC
> with `CorruptedTreeException` wrapping `Invalid version for inner update`:
> {code:java}
> java.lang.AssertionError: Invalid version for inner update [name=cache,
> topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], isNew=false,
> entry=GridDhtCacheEntry [..., ver=GridCacheVersion [topVer=394892252,
> order=1783412249558, nodeOrder=2, dataCenterId=0], ...]],
> newVer=GridCacheVersion [topVer=394892251, order=1783412250062, nodeOrder=1,
> dataCenterId=0]]
> at
> GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5385)
> at
> GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4738)
> ...
> at
> IgniteWalRebalanceTest.testRebalanceCancelOnSupplyError(IgniteWalRebalanceTest.java:509)
> {code}
> _Mechanism_
> GridCacheVersion comparison (GridCacheVersion.compareTo()) compares first by
> topVer, then by order. The assertion
> `ATOMIC_VER_COMPARATOR.compare(entry.ver, newVer) <= 0` fails because the new
> write's topVer (394892251) is less than the recovered entry's topVer
> (394892252).
> GridCacheVersion.topVer is computed as:
> {code:java}
> // GridCacheVersionManager.next():
> return next(topVer + offset, ...);
> // where:
> // topVer = readyTopologyVersion().topologyVersion() (from partition
> exchange)
> // offset = (gridStartTime - TOP_VER_BASE_TIME) / 1000 (grows by 1 per
> second) {code}
> In the test, after stopAllGrids() + startGrid(0):
> 1. readyTopologyVersion() resets to a small value (~1–3) after partition
> exchange completes
> 2. offset is recalculated from the new gridStartTime, which is only slightly
> larger than the previous one
> 3. Recovered data retains `topVer = prevReadyTopVer(~3) + prevOffset` from
> the previous cluster session
> 4. New writes receive `topVer = currReadyTopVer(~1) + newOffset`
> On slow TC machines, `newOffset - prevOffset < 1` (insufficient time passed
> between stop and start), so:
> {code:java}
> 394892252 = 3 + prevOffset (recovered entry)
> 394892251 = 1 + newOffset (new write)
> → 394892251 < 394892252 → assertion failure{code}
> On faster local machines, enough time passes between stopAllGrids() and
> startGrid(0) for offset to grow by ≥2 seconds, compensating for the lower
> readyTopologyVersion.
> *Fix*
> * *Keep the supplier (node 0) alive* — stop only nodes 1 and 2 — so
> {{topVer}} stays monotonic and recovered entries never outrank new writes.
> This mirrors {{testMultipleNodesFailHistoricalRebalance}} in the same file.
> * *Add {{spi.waitForBlocked()}}* before injecting the WAL-read failure. A
> warm supplier serves the historical rebalance fast enough that, without this
> barrier, the rebalance can finish before the failure fires and the future
> returns {{{}true{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
