[
https://issues.apache.org/jira/browse/IGNITE-25028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17957360#comment-17957360
]
Mirza Aliev commented on IGNITE-25028:
--------------------------------------
h3. What’s happening in the test from an external user’s point of view:
# start [A, B, C, D, E]
# create HA zone and table t1 (partition = 1)
# stop majority [B, C, D, E]
# insert data into table t1
*Result:* the insert fails with a sendWithRetry timeout.
h3. What’s happening under the hood:
When we stop those four nodes in step 3, the cluster loses majority. This
triggers the resetPartition mechanism, which writes a “force pending” state
containing only [A].
In the log you’ll see these three lines back-to-back:
{code:java}
1) [2025-05-22T20:19:01,928][INFO ] Partition 19_part_0 returned
PENDING_KEY_UPDATED status on reset attempt
2) [2025-05-22T20:19:01,928][INFO ] New leader elected. Going to apply new
configuration [tablePartitionId=19_part_0, peers=[A], learners=[]]
3) [2025-05-22T20:19:01,928][INFO ] Sending changePeersAndLearnersAsync request
for group=19_part_0 to peers=[A] and learners=[] with leader=A and term=1
{code}
# Line 1 shows that the new “force pending” [A] was recorded.
# Line 2 is the leader-election callback: it reads [A] and reconfigures to [A].
# Line 3 is the first changePeersAndLearnersAsync call, moving from [A, B, C,
D, E] to [A] under term 1.
Because this is so fast, there’s a race: Raft is told to shrink from five
replicas to one immediately.
{code:java}
[2025-05-22T20:19:03,140][ERROR] Fail to run ConfigurationChangeDone, status:
EPERM<1008>: Leader stepped down
{code}
Meanwhile, resetPeers logic is skipped:
{code:java}
[2025-05-22T20:19:01,930][WARN ] Reset peers for raft group 19_part_0 →
PeersAndLearners [peers=[A], learners=[]]
[2025-05-22T20:19:01,930][WARN ] Node 19_part_0/A set peers—needs to wait for
current conf change; old=[A,B,C,D,E], new=[A]
[2025-05-22T20:19:01,931][WARN ] Node 19_part_0/A refused concurrent config
change; old=A, new=A
{code}
Finally, when Raft retries sendWithRetry on the original
changePeersAndLearnersAsync, it fails because there’s no leader:
{code:java}
[2025-05-22T20:19:03,158][ERROR] handleRequest
ChangePeersAndLearnersAsyncRequestImpl … failed
java.lang.IllegalStateException: Not leader peerId = <19_part_0/A>
at org.apache.ignite.raft.jraft.core.NodeImpl.listPeers(NodeImpl.java:3381)
…
{code}
Without a valid leader, the group can’t re-form to [A, B, C, D, E], so the
client’s sendWithRetry ultimately times out.
To sum up: any concurrent or stale rebalance intent could interfere to
resetPartitions process and reset intent could be missed.
Solution: provide a retry logic for reset peers
https://issues.apache.org/jira/browse/IGNITE-25543
> ItHighAvailablePartitionsRecoveryTest.testScaleUpAfterHaRecoveryWhenMajorityLoss
> is flaky
> -----------------------------------------------------------------------------------------
>
> Key: IGNITE-25028
> URL: https://issues.apache.org/jira/browse/IGNITE-25028
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexander Lapin
> Assignee: Mirza Aliev
> Priority: Major
> Labels: ignite-3
> Attachments: _Integration_Tests_Module_Table_35000.log.zip
>
>
> {code:java}
> org.opentest4j.AssertionFailedError: Unexpected exception at
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42) at
> app//org.junit.jupiter.api.Assertions.fail(Assertions.java:150) at
> app//org.apache.ignite.internal.table.distributed.disaster.AbstractHighAvailablePartitionsRecoveryTest.insertValues(AbstractHighAvailablePartitionsRecoveryTest.java:506)
> at
> app//org.apache.ignite.internal.table.distributed.disaster.ItHighAvailablePartitionsRecoveryTest.testScaleUpAfterHaRecoveryWhenMajorityLoss(ItHighAvailablePartitionsRecoveryTest.java:269)
> at [email protected]/java.lang.reflect.Method.invoke(Method.java:568) at
> [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511) at
> [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)Caused by:
> java.lang.AssertionError: java.util.concurrent.ExecutionException:
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535
> TraceId:10db50e8-93b6-42cd-9d80-7b0a628f6e2d Send with retry timed out
> [retryCount = 50, groupId = 19_part_1, traceId =
> b147d859-f7ed-470b-b096-c7ad198374c7, request =
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
> originCommand = null]. at
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
> at
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
> at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at
> org.apache.ignite.internal.table.distributed.disaster.AbstractHighAvailablePartitionsRecoveryTest.insertValues(AbstractHighAvailablePartitionsRecoveryTest.java:492)
> ... 4 more {code}
> [TC
> link|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/9011597]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)