CRZbulabula opened a new pull request, #18275:
URL: https://github.com/apache/iotdb/pull/18275
## Description
Failed Region replica creation can leave persisted `RegionCreateTask`s in
the ConfigNode maintenance queue. Those tasks could race with `DROP DATABASE`,
survive leader changes or snapshots, and issue delayed create RPCs after the
RegionGroup had been removed. Large retry queues could also amplify RPC load
and direct-memory pressure, while partial DataNode creation failures could
leave local Region, Consensus/WAL, executor, metric, or memory state behind.
This PR keeps `RemoveRegionGroupProcedure` as the deletion mechanism and
completes the lifecycle of Region creation tasks across ConfigNode and DataNode.
### ConfigNode lifecycle and deletion fencing
- Add an idempotent, persisted `BatchRemoveRegionCreateTasksPlan` that
removes every queued task matching a set of RegionIds and is replayed through
ConfigNode Consensus and snapshots.
- Make `DeleteDatabaseProcedure` pre-delete the database, exclude its
Regions from maintenance, durably clear all matching create tasks, wait for
in-flight create batches, submit `RemoveRegionGroupProcedure`, and only then
remove the database partition table.
- Serialize DROP and Region maintenance with a shared lifecycle lock so a
task removed from the queue cannot still recreate a Region after deletion.
- Revalidate every task immediately before dispatch: the database must exist
and not be pre-deleted, the Region must still belong to it, the replica set
must still contain the target DataNode, and the replica must still be missing.
Invalid tasks are durably removed.
- Repair stale maintenance tasks when a ConfigNode becomes leader, including
orphaned tasks left by older versions.
- Fence delayed create RPCs after RegionGroup deletion starts.
### Initial RegionGroup creation race
This PR incorporates and supersedes #18273. It adds database lifecycle
generations, database-level procedure locks, atomic `CreateRegionGroupsPlan`
validation, snapshot/procedure compatibility, and compensation cleanup. This
prevents both initial `CreateRegionGroupsProcedure` work and later
`RegionCreateTask` retries from binding to a dropped or same-name recreated
database.
Legacy Consensus logs remain replayable. Recovered generation-less
procedures are restricted to the legacy database generation so they cannot
create Regions in a database incarnation created by the upgraded version.
### Retry and resource control
- Limit each maintenance batch per DataNode to 32 SchemaRegions and 64
DataRegions.
- Attempt each task once per scheduler cycle instead of performing six
immediate retries.
- Add exponential backoff with jitter, per-node OOM cooldown, and bounded
in-flight create RPCs.
### Transactional DataNode creation
- Commit a newly created DataRegion only after local Region and
WAL/Consensus initialization both succeed.
- Roll back newly allocated Region, Consensus/WAL state, metrics, executors,
and direct-memory budget on failure.
- Apply equivalent rollback behavior to SchemaRegion creation.
- Distinguish a Region created by the current request from a pre-existing
Region so idempotent retries never remove healthy state.
## Tests
- ConfigNode focused tests: 126 tests, 0 failures.
- `DataNodeRegionManagerTest`: 5 tests, 0 failures.
- Full English reactor `test-compile`: 50/50 modules successful.
- Full Chinese locale reactor `test-compile`: 50/50 modules successful.
- Spotless, Checkstyle, and `git diff --check` passed.
The focused tests cover persisted batch cancellation, multiple failed
replicas for one Region, concurrent maintenance and DROP, leader/snapshot
recovery, same-name database recreation, lifecycle generation compatibility,
compensation cleanup, and DataNode rollback after WAL/direct-memory or Ratis
failures.
This PR has:
- [x] been self-reviewed.
- [x] concurrent write
- [x] added comments explaining non-obvious concurrency and recovery
behavior.
- [x] added or updated unit tests for the new code paths.
- [ ] added integration tests.
- [ ] been tested in a fault-injected 3C3D cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
