CRZbulabula opened a new pull request, #18981:
URL: https://github.com/apache/iotdb/pull/18981
## Problem
`IoTDBRegionGroupLeaderDistributionIT` fails intermittently in the cluster
IT suites. In one
1C3D run all three of its test methods failed at once:
```
IoTDBRegionGroupLeaderDistributionIT.testBasicLeaderDistribution:135
expected:<3> but was:<2>
IoTDBRegionGroupLeaderDistributionIT.testCFDWithUnknownStatus:205
IoTDBRegionGroupLeaderDistributionIT.testCFDWithReadOnlyStatus:321
```
## Root cause
Both problems are in the assertions, not in the balancer.
**1. `testBasicLeaderDistribution` sampled the distribution exactly once**,
immediately after
`getOrCreateDataPartitionTable`. The RegionGroups have only just been
created at that point, so
the leaders are still being elected and `RouteBalancer` is still migrating
them. From the
ConfigNode log of the failing run, while all three DataNodes were already
`Running`:
```
20:59:40,114 [LeaderBalancer] Try to change the leader of Region:
TConsensusGroupId(type:SchemaRegion, id:0) to DataNode: 3
20:59:40,317 [LeaderBalancer] Try to change the leader of Region:
TConsensusGroupId(type:DataRegion, id:1) to DataNode: 3
20:59:43,xxx testBasicLeaderDistribution -> Assert.assertEquals(3,
leaderCounter.size()) // was 2
```
The single sample therefore observed a transient state, and a moment later
the distribution was
fine.
**2. `testCFDWithUnknownStatus` and `testCFDWithReadOnlyStatus` did retry**
(50 x 1s), but
asserted a fixed per-DataNode leader count (`databaseNum /
TEST_DATA_NODE_NUM`, i.e. "exactly 2
leaders each"). That is not a state the leader balancer guarantees:
- it balances the System and Audit databases as well, while these assertions
filter those rows
out, so the counted subset is not the quantity being balanced;
- the number of RegionGroups is not necessarily divisible by the number of
DataNodes.
In the observed run the cluster held 5 RegionGroups for 3 databases (4
DataRegionGroups + 1
SchemaRegionGroup); the Audit database owns the extra DataRegionGroup and
the only
SchemaRegionGroup. Filtering the Audit database therefore leaves a subset
whose even split is not
what the balancer optimizes, so the loop could not converge and failed only
after the entire retry
budget.
## Fix
- Extract the leader counting into `countRegionLeaders` and the waiting into
`awaitBalancedLeaderDistribution`, replacing four copies of the same
sampling block.
- **Poll** instead of sampling once.
- Assert the property the balancer actually provides: every expected
DataNode holds at least one
RegionGroup leader, and no two DataNodes differ by more than one leader.
This is the invariant
`IoTDBMultiDBRegionGroupLeaderDistributionIT` already uses.
- Report the observed distribution in the failure message instead of a bare
`AssertionError`.
- The post-shutdown check reuses the helper with `TEST_DATA_NODE_NUM - 1`,
so it additionally
verifies that the leaders of the stopped DataNode really migrated: an
unassigned leader keeps
the count of leader-holding DataNodes below the expected size, which a
fixed per-node count
never checked.
For a converged cluster the behaviour is unchanged: 3 DataNodes holding 3
RegionGroups still has
to be 1/1/1.
## Verification
- `mvn test-compile -pl integration-test -am -DskipTests -P
with-integration-tests` — BUILD
SUCCESS (39 modules).
- `mvn spotless:apply -pl integration-test -P with-integration-tests` — no
diff.
- Reproduced the assertions against the failing CI run's cluster logs to
confirm the retry budget
was being spent waiting for a state that cannot be reached, rather than
timing out on a merely
slow balancer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]