CRZbulabula opened a new pull request, #18981:
URL: https://github.com/apache/iotdb/pull/18981

   ## Problem
   
   `IoTDBRegionGroupLeaderDistributionIT` fails intermittently in the cluster 
IT suites. In one
   1C3D run all three of its test methods failed at once:
   
   ```
   IoTDBRegionGroupLeaderDistributionIT.testBasicLeaderDistribution:135 
expected:<3> but was:<2>
   IoTDBRegionGroupLeaderDistributionIT.testCFDWithUnknownStatus:205
   IoTDBRegionGroupLeaderDistributionIT.testCFDWithReadOnlyStatus:321
   ```
   
   ## Root cause
   
   Both problems are in the assertions, not in the balancer.
   
   **1. `testBasicLeaderDistribution` sampled the distribution exactly once**, 
immediately after
   `getOrCreateDataPartitionTable`. The RegionGroups have only just been 
created at that point, so
   the leaders are still being elected and `RouteBalancer` is still migrating 
them. From the
   ConfigNode log of the failing run, while all three DataNodes were already 
`Running`:
   
   ```
   20:59:40,114 [LeaderBalancer] Try to change the leader of Region: 
TConsensusGroupId(type:SchemaRegion, id:0) to DataNode: 3
   20:59:40,317 [LeaderBalancer] Try to change the leader of Region: 
TConsensusGroupId(type:DataRegion, id:1) to DataNode: 3
   20:59:43,xxx testBasicLeaderDistribution -> Assert.assertEquals(3, 
leaderCounter.size()) // was 2
   ```
   
   The single sample therefore observed a transient state, and a moment later 
the distribution was
   fine.
   
   **2. `testCFDWithUnknownStatus` and `testCFDWithReadOnlyStatus` did retry** 
(50 x 1s), but
   asserted a fixed per-DataNode leader count (`databaseNum / 
TEST_DATA_NODE_NUM`, i.e. "exactly 2
   leaders each"). That is not a state the leader balancer guarantees:
   
   - it balances the System and Audit databases as well, while these assertions 
filter those rows
     out, so the counted subset is not the quantity being balanced;
   - the number of RegionGroups is not necessarily divisible by the number of 
DataNodes.
   
   In the observed run the cluster held 5 RegionGroups for 3 databases (4 
DataRegionGroups + 1
   SchemaRegionGroup); the Audit database owns the extra DataRegionGroup and 
the only
   SchemaRegionGroup. Filtering the Audit database therefore leaves a subset 
whose even split is not
   what the balancer optimizes, so the loop could not converge and failed only 
after the entire retry
   budget.
   
   ## Fix
   
   - Extract the leader counting into `countRegionLeaders` and the waiting into
     `awaitBalancedLeaderDistribution`, replacing four copies of the same 
sampling block.
   - **Poll** instead of sampling once.
   - Assert the property the balancer actually provides: every expected 
DataNode holds at least one
     RegionGroup leader, and no two DataNodes differ by more than one leader. 
This is the invariant
     `IoTDBMultiDBRegionGroupLeaderDistributionIT` already uses.
   - Report the observed distribution in the failure message instead of a bare 
`AssertionError`.
   - The post-shutdown check reuses the helper with `TEST_DATA_NODE_NUM - 1`, 
so it additionally
     verifies that the leaders of the stopped DataNode really migrated: an 
unassigned leader keeps
     the count of leader-holding DataNodes below the expected size, which a 
fixed per-node count
     never checked.
   
   For a converged cluster the behaviour is unchanged: 3 DataNodes holding 3 
RegionGroups still has
   to be 1/1/1.
   
   ## Verification
   
   - `mvn test-compile -pl integration-test -am -DskipTests -P 
with-integration-tests` — BUILD
     SUCCESS (39 modules).
   - `mvn spotless:apply -pl integration-test -P with-integration-tests` — no 
diff.
   - Reproduced the assertions against the failing CI run's cluster logs to 
confirm the retry budget
     was being spent waiting for a state that cannot be reached, rather than 
timing out on a merely
     slow balancer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to