CRZbulabula opened a new pull request, #18033: URL: https://github.com/apache/iotdb/pull/18033
## Problem The ConfigNode-leader periodically drained a persistent **RegionMaintainer queue** (`PartitionManager.maintainRegionReplicas`, every 10s) by sending **synchronous** `deleteRegion` RPCs to DataNodes. For a data region this deletion can be slow (removing TsFiles + the consensus peer) and may **outlive the RPC timeout**. When it does: 1. The ConfigNode times out and retries the same task 10s later. 2. On the DataNode the **second** call re-runs the deletion, which now returns `SUCCESS` because the region is already gone (`ConsensusGroupNotExistException` is swallowed and deleting an already-removed region is a no-op). 3. The ConfigNode treats that `SUCCESS` as completion, polls the task off the queue, and forgets it — **while the first call is still running** on the DataNode. The result is a *ghost task*: the cluster keeps executing a deletion the coordinator has already discarded. ## Approach Replace the whole RegionMaintainer queue with **Procedures**, mirroring the existing `AddRegionPeerProcedure` / `RemoveRegionPeerProcedure` pattern (`RegionOperationProcedure<State>` + `submit*Task(procId, …)` + `waitTaskFinish` polling). Both the CREATE and DELETE paths move to dedicated procedures, gaining persistence and retry across leader changes. ### DataNode — async delete - New `deleteRegionAsync(TMaintainPeerReq)` RPC on the internal service. - New `DeleteRegionTask` in `RegionMigrateService`: runs the deletion in the background and records its terminal state in the existing `taskResultMap`, so the ConfigNode polls `getRegionMaintainResult` for progress (`PROCESSING` / `SUCCESS` / `FAIL` / `TASK_NOT_EXIST`) instead of trusting a single RPC response. The task **returns immediately after `taskFail` and never falls through to `taskSucceed`**, so a failed deletion can never be reported as success. Retried submits are deduped by `taskId`. ### ConfigNode — procedures - `DeleteRegionProcedure`: async submit + `waitTaskFinish` poll, `taskId = procId`, idempotent on retry / leader change (submit only on first entry; the DataNode dedups by `taskId`). - `CreateRegionProcedure`: region creation already returns a final, unambiguous status from a single RPC (no ghost-task race), so it stays **synchronous** and the procedure only adds persistence and retry. - `DeleteDatabaseProcedure` and `CreateRegionGroupsProcedure` now spawn these as child procedures instead of offering to the queue, and **every completed maintain task is logged** (region, DataNode, duration). - `PartitionManager.maintainRegionReplicas` and the `RegionCleaner` scheduler are removed. ### Upgrade safety (deprecate, don't delete) - The `OfferRegionMaintainTasks` / `PollRegionMaintainTask` / `PollSpecificRegionMaintainTask` plan types and classes are kept (`@Deprecated`, still deserialized and applied as no-ops) so an old ConfigNode consensus log still replays. - The `PartitionInfo` snapshot has no version header, so the trailing RegionMaintainer block is still consumed on load — its tasks are discarded (with a WARN naming them so any loss is observable) — and written as empty on save, keeping the byte layout unchanged. ## Tests - `DeleteRegionProcedureTest` / `CreateRegionProcedureTest` — serialization round-trip through `ProcedureFactory` (exercises the new `ProcedureType` codes + factory registration). - `PartitionInfoTest.testLoadLegacySnapshotWithRegionMaintainTasksIsDrained` — a snapshot written with pending tasks loads cleanly under the new code and ends with an empty maintain list. - Existing `ConfigPhysicalPlanSerDeTest` for the retained plan types stays green (proves consensus-log replay compatibility). All confignode + datanode unit tests above pass; `spotless:check` and `checkstyle` are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
