[
https://issues.apache.org/jira/browse/IGNITE-28832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-28832:
-----------------------------------
Description:
StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
Success Rate: 21.7% (Last 106 Runs) 83 failed, 23 successful
[https://ci2.ignite.apache.org/test/2355900617476832260?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
*1. Failure reason (stacktrace from master log
_Apache_Ignite_2.x_Tests_Queries_3_17496.log)*
The test fails in IgniteBinaryCacheQueryTestSuite3 with an exchange timeout
(default 30 s). Stacktrace:
{code:java}
class org.apache.ignite.IgniteException: Timeout of waiting for topology
map update [igniteInstanceName=stat.StatisticsConfigurationTest1,
cache=SMALLnull, cacheId=1165137646,
topVer=AffinityTopologyVersion [topVer=7, minorTopVer=2], p=0,
readVer=AffinityTopologyVersion [topVer=7, minorTopVer=2],
locNode=TcpDiscoveryNode [id=311d5634-29a9-41fe-83e1-744048a00001,
...]]
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:806)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:599)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:584)
at
org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.stopGridAndChangeBaseline(StatisticsConfigurationTest.java:160)
at
org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.updateStatisticsOnChangeTopology(StatisticsConfigurationTest.java:300)
...{code}
The test passes locally on fast machines and only flakes on slow CI nodes.
{*}2. Root cause{*}{*}{*}
updateStatisticsOnChangeTopology() repeatedly stops/restarts server nodes and
(in persist=true mode) manually adjusts the baseline via
cluster().setBaselineTopology(...).
After a server node leaves the baseline, the partitions it owned become *LOST*
(log confirms Detected lost partitions [grp=SMALLnull, parts=[...],
topVer=7.0]). The table is created with BACKUPS=1 and default
PartitionLossPolicy = READ_ONLY_SAFE. Because partition-loss detection runs
during the partition exchange _before_ a surviving backup is promoted to
primary, partitions whose owner left are marked LOST instead of being
rebalanced.
LOST partitions have {*}no owners{*}, so awaitPartitionMapExchange() — which
waits until the topology is fully rebalanced (rebalanced=true) — can never
converge and times out after 30 s. Hence the intermittent failure regardless of
actual machine slowness.
*3. The fix*
Instead of manually re-setting the baseline after each node start/stop (which
is what makes partitions become LOST), the baseline is set once to *auto-adjust
with zero timeout* at cluster creation. This makes the baseline update
automatically whenever nodes join or leave, so backups are promoted to
primaries _before_ detectLostPartitions runs during the exchange — partitions
no longer become LOST and the rebalance completes.
The exchange timeout is also increased (×3 of the default) to tolerate slow CI
machines, since rebalancing a persistent cache can take longer than the default
30 s.
*Validation note:* after this fix the test ran 30 times on CI and failed only
once, and that single failure was a pure 30 s timeout on rebalanced=false (no
LOST partitions) caused by the default exchange timeout — addressed by raising
getPartitionMapExchangeTimeout().
was:
StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
Success Rate: 21.7% (Last 106 Runs) 83 failed, 23 successful
[https://ci2.ignite.apache.org/test/2355900617476832260?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
*1. Failure reason (stacktrace from master log
_Apache_Ignite_2.x_Tests_Queries_3_17496.log)*
The test fails in IgniteBinaryCacheQueryTestSuite3 with an exchange timeout
(default 30 s). Stacktrace:
{code:java}
class org.apache.ignite.IgniteException: Timeout of waiting for topology
map update [igniteInstanceName=stat.StatisticsConfigurationTest1,
cache=SMALLnull, cacheId=1165137646,
topVer=AffinityTopologyVersion [topVer=7, minorTopVer=2], p=0,
readVer=AffinityTopologyVersion [topVer=7, minorTopVer=2],
locNode=TcpDiscoveryNode [id=311d5634-29a9-41fe-83e1-744048a00001,
...]]
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:806)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:599)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:584)
at
org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.stopGridAndChangeBaseline(StatisticsConfigurationTest.java:160)
at
org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.updateStatisticsOnChangeTopology(StatisticsConfigurationTest.java:300)
...{code}
The test passes locally on fast machines and only flakes on slow CI nodes.
{*}2. Root cause{*}{*}{{*}}
updateStatisticsOnChangeTopology() repeatedly stops/restarts server nodes and
(in persist=true mode) manually adjusts the baseline via
cluster().setBaselineTopology(...).
After a server node leaves the baseline, the partitions it owned become *LOST*
(log confirms Detected lost partitions [grp=SMALLnull, parts=[...],
topVer=7.0]). The table is created with BACKUPS=1 and default
PartitionLossPolicy = READ_ONLY_SAFE. Because partition-loss detection runs
during the partition exchange _before_ a surviving backup is promoted to
primary, partitions whose owner left are marked LOST instead of being
rebalanced.
LOST partitions have {*}no owners{*}, so awaitPartitionMapExchange() — which
waits until the topology is fully rebalanced (rebalanced=true) — can never
converge and times out after 30 s. Hence the intermittent failure regardless of
actual machine slowness.
*3. The fix*
Instead of manually re-setting the baseline after each node start/stop (which
is what makes partitions become LOST), the baseline is set once to *auto-adjust
with zero timeout* at cluster creation. This makes the baseline update
automatically whenever nodes join or leave, so backups are promoted to
primaries _before_ detectLostPartitions runs during the exchange — partitions
no longer become LOST and the rebalance completes.
The exchange timeout is also increased (×3 of the default) to tolerate slow CI
machines, since rebalancing a persistent cache can take longer than the default
30 s.
*Validation note:* after this fix the test ran 30 times on CI and failed only
once, and that single failure was a pure 30 s timeout on rebalanced=false (no
LOST partitions) caused by the default exchange timeout — addressed by raising
getPartitionMapExchangeTimeout().
> Fix flaky
> StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
> ------------------------------------------------------------------------------------
>
> Key: IGNITE-28832
> URL: https://issues.apache.org/jira/browse/IGNITE-28832
> Project: Ignite
> Issue Type: Test
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain, ise
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
> Success Rate: 21.7% (Last 106 Runs) 83 failed, 23 successful
> [https://ci2.ignite.apache.org/test/2355900617476832260?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
>
> *1. Failure reason (stacktrace from master log
> _Apache_Ignite_2.x_Tests_Queries_3_17496.log)*
> The test fails in IgniteBinaryCacheQueryTestSuite3 with an exchange timeout
> (default 30 s). Stacktrace:
> {code:java}
> class org.apache.ignite.IgniteException: Timeout of waiting for
> topology map update [igniteInstanceName=stat.StatisticsConfigurationTest1,
> cache=SMALLnull, cacheId=1165137646,
> topVer=AffinityTopologyVersion [topVer=7, minorTopVer=2], p=0,
> readVer=AffinityTopologyVersion [topVer=7, minorTopVer=2],
> locNode=TcpDiscoveryNode [id=311d5634-29a9-41fe-83e1-744048a00001,
> ...]]
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:806)
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:599)
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:584)
> at
> org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.stopGridAndChangeBaseline(StatisticsConfigurationTest.java:160)
> at
> org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.updateStatisticsOnChangeTopology(StatisticsConfigurationTest.java:300)
> ...{code}
> The test passes locally on fast machines and only flakes on slow CI nodes.
> {*}2. Root cause{*}{*}{*}
> updateStatisticsOnChangeTopology() repeatedly stops/restarts server nodes and
> (in persist=true mode) manually adjusts the baseline via
> cluster().setBaselineTopology(...).
> After a server node leaves the baseline, the partitions it owned become
> *LOST* (log confirms Detected lost partitions [grp=SMALLnull, parts=[...],
> topVer=7.0]). The table is created with BACKUPS=1 and default
> PartitionLossPolicy = READ_ONLY_SAFE. Because partition-loss detection runs
> during the partition exchange _before_ a surviving backup is promoted to
> primary, partitions whose owner left are marked LOST instead of being
> rebalanced.
> LOST partitions have {*}no owners{*}, so awaitPartitionMapExchange() — which
> waits until the topology is fully rebalanced (rebalanced=true) — can never
> converge and times out after 30 s. Hence the intermittent failure regardless
> of actual machine slowness.
> *3. The fix*
> Instead of manually re-setting the baseline after each node start/stop (which
> is what makes partitions become LOST), the baseline is set once to
> *auto-adjust with zero timeout* at cluster creation. This makes the baseline
> update automatically whenever nodes join or leave, so backups are promoted to
> primaries _before_ detectLostPartitions runs during the exchange — partitions
> no longer become LOST and the rebalance completes.
> The exchange timeout is also increased (×3 of the default) to tolerate slow
> CI machines, since rebalancing a persistent cache can take longer than the
> default 30 s.
> *Validation note:* after this fix the test ran 30 times on CI and failed only
> once, and that single failure was a pure 30 s timeout on rebalanced=false (no
> LOST partitions) caused by the default exchange timeout — addressed by
> raising getPartitionMapExchangeTimeout().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
