CRZbulabula opened a new pull request, #17934:
URL: https://github.com/apache/iotdb/pull/17934
## Problem
When a `remove datanode` is in progress, the ConfigNode could still allocate
brand-new Region replicas onto the DataNode being removed.
This is especially likely when the target DataNode was killed (e.g. `kill
-9`) before the removal: the failure detector reports such a node as `Unknown`
rather than `Removing`, and `RegionBalancer` **intentionally** keeps `Unknown`
DataNodes as allocation candidates (to cope with an insufficient number of
online nodes). As a result, a new region (e.g. for the internal `root.__system`
database) could be assigned a replica on the dead node. That replica can never
be created there (`Connection refused`), yet the metadata keeps the assignment
and retries forever, so the `RemoveDataNodesProcedure` gets stuck and the
target DataNode never disappears from `show datanodes`.
Observed timeline (from the report):
```
11:31:06.210 Submit RemoveDataNodesProcedure successfully
11:31:06.550 create SchemaRegion 8 on DataNodes [3, 7, 6] # DN7 is being
removed
11:31:07.444 create DataRegion 9 on DataNodes [5, 7] # DN7 is being
removed
Failed to CREATE_*_REGION on DataNode 7: Connection refused
(retried forever)
```
## Root cause
`RegionBalancer.genRegionGroupsAllocationPlan` gathers allocation candidates
with:
```java
getNodeManager().filterDataNodeThroughStatus(NodeStatus.Running,
NodeStatus.Unknown)
```
A node-status filter alone cannot fix this: a DataNode killed before removal
is `Unknown` (not `Removing`), because
`DataNodeHeartbeatCache.updateCurrentStatistics` lets the failure detector
override the `Removing` status back to `Unknown` once heartbeats stop. So the
removing node still passes the filter.
## Fix
Consult the in-progress `RemoveDataNodesProcedure` — the authoritative,
leader-switch-durable source of which DataNodes are being removed — and exclude
those nodes from the allocation candidates.
- Add `ProcedureManager.getRemovingDataNodeIds()`, which scans the
unfinished `RemoveDataNodesProcedure`(s) and returns the removing DataNode ids.
This mirrors the existing pattern in `ProcedureManager.checkRemoveDataNodes` /
`checkRegionOperationWithRemoveDataNode`.
- `RegionBalancer.genRegionGroupsAllocationPlan` now filters those ids out
of the candidate list, in the same spirit as the existing
`RemoveDataNodeHandler.selectedRegionMigrationPlans`.
This keeps the existing "allow `Unknown` candidates when online nodes are
scarce" behavior intact, and only removes nodes that are actively being
removed. The removal procedure already guarantees
(`checkEnoughDataNodeAfterRemoving`) that enough non-removing nodes remain, so
this never spuriously triggers `NotEnoughDataNodeException`.
## Test
`IoTDBRemoveDataNodeRegionAllocationIT` kills a DataNode that hosts regions,
submits the removal, and — while the removal is still in progress — forces a
fresh Region allocation by creating a new database. It asserts that none of the
**newly allocated** regions land on the DataNode being removed (comparing
against a pre-allocation snapshot of region ids, since the removing node
legitimately keeps hosting its own pre-existing regions until each finishes
migrating away), and that the removal then completes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
