[ 
https://issues.apache.org/jira/browse/IGNITE-25028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17957360#comment-17957360
 ] 

Mirza Aliev commented on IGNITE-25028:
--------------------------------------

h3. What’s happening in the test from an external user’s point of view:
 # start [A, B, C, D, E]
 # create HA zone and table t1 (partition = 1)
 # stop majority [B, C, D, E]
 # insert data into table t1

*Result:* the insert fails with a sendWithRetry timeout.
h3. What’s happening under the hood:

When we stop those four nodes in step 3, the cluster loses majority. This 
triggers the resetPartition mechanism, which writes a “force pending” state 
containing only [A].

In the log you’ll see these three lines back-to-back:
{code:java}
1) [2025-05-22T20:19:01,928][INFO ] Partition 19_part_0 returned 
PENDING_KEY_UPDATED status on reset attempt
2) [2025-05-22T20:19:01,928][INFO ] New leader elected. Going to apply new 
configuration [tablePartitionId=19_part_0, peers=[A], learners=[]]
3) [2025-05-22T20:19:01,928][INFO ] Sending changePeersAndLearnersAsync request 
for group=19_part_0 to peers=[A] and learners=[] with leader=A and term=1
{code}
 # Line 1 shows that the new “force pending” [A] was recorded.
 # Line 2 is the leader-election callback: it reads [A] and reconfigures to [A].
 # Line 3 is the first changePeersAndLearnersAsync call, moving from [A, B, C, 
D, E] to [A] under term 1.

Because this is so fast, there’s a race: Raft is told to shrink from five 
replicas to one immediately.
{code:java}
[2025-05-22T20:19:03,140][ERROR] Fail to run ConfigurationChangeDone, status: 
EPERM<1008>: Leader stepped down
{code}
Meanwhile, resetPeers logic is skipped:
{code:java}
[2025-05-22T20:19:01,930][WARN ] Reset peers for raft group 19_part_0 → 
PeersAndLearners [peers=[A], learners=[]]
[2025-05-22T20:19:01,930][WARN ] Node 19_part_0/A set peers—needs to wait for 
current conf change; old=[A,B,C,D,E], new=[A]
[2025-05-22T20:19:01,931][WARN ] Node 19_part_0/A refused concurrent config 
change; old=A, new=A
{code}
Finally, when Raft retries sendWithRetry on the original 
changePeersAndLearnersAsync, it fails because there’s no leader:
{code:java}
[2025-05-22T20:19:03,158][ERROR] handleRequest 
ChangePeersAndLearnersAsyncRequestImpl … failed
java.lang.IllegalStateException: Not leader peerId = <19_part_0/A>
    at org.apache.ignite.raft.jraft.core.NodeImpl.listPeers(NodeImpl.java:3381)
    …
{code}
Without a valid leader, the group can’t re-form to [A, B, C, D, E], so the 
client’s sendWithRetry ultimately times out.

To sum up: any concurrent or stale rebalance intent could interfere to 
resetPartitions process and reset intent could be missed.

Solution: provide a retry logic for reset peers 
https://issues.apache.org/jira/browse/IGNITE-25543

> ItHighAvailablePartitionsRecoveryTest.testScaleUpAfterHaRecoveryWhenMajorityLoss
>  is flaky
> -----------------------------------------------------------------------------------------
>
>                 Key: IGNITE-25028
>                 URL: https://issues.apache.org/jira/browse/IGNITE-25028
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Alexander Lapin
>            Assignee: Mirza Aliev
>            Priority: Major
>              Labels: ignite-3
>         Attachments: _Integration_Tests_Module_Table_35000.log.zip
>
>
> {code:java}
> org.opentest4j.AssertionFailedError: Unexpected exception  at 
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42)  at 
> app//org.junit.jupiter.api.Assertions.fail(Assertions.java:150)  at 
> app//org.apache.ignite.internal.table.distributed.disaster.AbstractHighAvailablePartitionsRecoveryTest.insertValues(AbstractHighAvailablePartitionsRecoveryTest.java:506)
>   at 
> app//org.apache.ignite.internal.table.distributed.disaster.ItHighAvailablePartitionsRecoveryTest.testScaleUpAfterHaRecoveryWhenMajorityLoss(ItHighAvailablePartitionsRecoveryTest.java:269)
>   at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)  at 
> [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)  at 
> [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)Caused by: 
> java.lang.AssertionError: java.util.concurrent.ExecutionException: 
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:10db50e8-93b6-42cd-9d80-7b0a628f6e2d Send with retry timed out 
> [retryCount = 50, groupId = 19_part_1, traceId = 
> b147d859-f7ed-470b-b096-c7ad198374c7, request = 
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
>  originCommand = null].  at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
>   at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
>   at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83)  at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10)  at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)  at 
> org.apache.ignite.internal.table.distributed.disaster.AbstractHighAvailablePartitionsRecoveryTest.insertValues(AbstractHighAvailablePartitionsRecoveryTest.java:492)
>   ... 4 more {code}
> [TC 
> link|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/9011597]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to