Caideyipi opened a new pull request, #19241: URL: https://github.com/apache/iotdb/pull/19241
## Description A replicated LOAD PREPARE or COMMIT can arrive before a missing PIECE is retried after reconnect. Returning a permanent LOAD error lets consensus retire the command before the replica has finished the task. This draft retries only temporarily incomplete replicated commands and preserves completed task identities across retries and restarts. **This is an incomplete fix and is not ready to merge.** The remaining recovery requirements below belong to the same LOAD retry path. Some underlying behaviors predate this patch, but the retry guarantee still depends on addressing them. ### Changes currently implemented - Introduce `LoadFileNotReadyException` and map it to `WRITE_PROCESS_REJECT` for replicated PREPARE/COMMIT only. Permanent failures and local command failures retain their existing error classification. - Persist COMMIT/ABORT outcomes outside task staging, include them in LOAD snapshots, and use the bounded in-memory history as a cache. Record empty ABORT outcomes and ignore late replicated PIECEs for completed tasks. - Coordinate staging snapshot copies with COMMIT/ABORT. Migrate legacy terminal records before retention listener registration, preserve staging when migration fails, and fence lazy manager initialization against region deletion. - Correct the lazy-initialization lock order relative to snapshot flushing. ### TODO before marking ready, and how to adapt 1. [ ] **Preserve acknowledged, unfinished staging.** Adapt `LoadTsFileManager.stop`, `TsFileWriterManager.close`, and `LoadTsFileCleaner.reclaimIfIdle` to distinguish coordinator temporary files from consensus-owned accepted state. Ordinary shutdown must close handles while preserving recoverable payload and progress. Idle time alone must not delete accepted replicated data: remote requests use a negative local search index, so the existing watermark guard does not protect them. Reclamation needs an explicit terminal decision, a proven safe replay frontier, or an implemented payload reconstruction protocol. Otherwise, a sender that already retired the PIECE batch can only retry PREPARE/COMMIT against missing data indefinitely. 2. [ ] **Make COMMIT import recoverable and idempotent.** Adapt `LoadTsFileManager.loadAll`, `TsFileWriterManager.loadAll`, and the `DataRegion` import path. Persist an import plan/COMMITTING state with stable per-task, per-partition installation identities and enough information to recognize published files after restart. Recovery must finish missing installations without reinstalling completed ones or repeating their publication effects. The current sequence imports files before persisting COMMIT, while ordinary import allocates a new version/name each time. Keeping the staged source does not by itself make replay safe. Moving the final COMMIT record before import would leave a different missing-data window and is insufficient. 3. [ ] **Coordinate the terminal decision with replication WAL durability.** Adapt the terminal-hit paths and WAL logging in `LoadTsFileManager.loadAll`/`deleteAll`. Currently the terminal record can become durable before the command reaches WAL, and a subsequent terminal-hit return skips WAL logging. Define a replayable decision/intent and local application-completion protocol so successful retries also satisfy outstanding replication obligations. An in-memory dispatcher re-offer is insufficient across restart or queue overflow. The design must reconcile stable decision identity, WAL replay, and application progress; a local terminal file alone must not authorize success when durable replication information is missing. 4. [ ] **Apply one state/ownership model to snapshots and lifecycle operations.** A bounded direction is a region-local task protocol such as RECEIVING -> PREPARED -> COMMITTING -> COMMITTED, with explicit permitted ABORTED transitions. Define the durability and acknowledgement conditions for every transition before implementation. Capture data files, active task/import state, terminal outcomes, and replay position from a consistent boundary. Preserve the existing staging-copy boundary, migration-before-listener ordering, and consistent lock order. Separate ordinary stop from region deletion. Terminal-history garbage collection needs a proven replay frontier; arbitrary TTL or an in-memory receive index is insufficient. Measure terminal-record growth and staging/snapshot copy latency. This does not require adding cross-region atomic LOAD or rollback semantics. 5. [ ] **Validate the recovery boundaries above.** Add failure-oriented coverage for acknowledged PIECE -> normal stop/restart -> later PREPARE/COMMIT; acknowledged PIECE -> idle reclamation attempt -> later command; partial/full import -> process termination before terminal publication -> restart/retry; terminal publication -> WAL failure/restart or dispatch queue overflow -> replica catch-up; and snapshots spanning these transitions. Assert installed-file identity/count, relevant publication effects, recovered task state, and replica catch-up rather than only successful status codes. Re-run the focused tests and the restart ClusterIT on the completed implementation, then perform a complete independent review before marking ready. ### Validation and revision limits - Current candidate: `a7d199b65162c9d62bad1c4e64883e33472fc58f`. - Standard focused local Maven reactor: **80 tests passed**, with zero failures, errors, or skips. Coverage includes state-machine classification, LOAD manager/snapshot/cleaner behavior, selected DataRegion lifecycle tests, and selected IoTDB snapshot tests. All 14 test methods added in this diff were run locally. - Spotless, Checkstyle, forbidden-API checks, and `git diff --check` passed. - These results do not establish crash safety for the unfinished TODOs above. No current-candidate ClusterIT, full CI, Chinese-locale compilation, process-kill crash test, or performance benchmark has been run. - The candidate was developed and tested on recorded baseline `6e7448d9bebd644259b695dfc8774d3d9f91252e`. The refreshed target `master` was `54bff3d9a421f9ba148fda3a26626df570422f54` when this draft was prepared. The candidate was not rebased, and an integration merge with that newer target has not been validated. - Review findings remain outstanding at the design level; this draft has not passed an issue-free independent review. This PR has: - [x] Added regression unit tests and run every newly added test locally. - [ ] Completed the recovery protocol TODOs above. - [ ] Validated the completed candidate with the restart ClusterIT. - [ ] Passed a complete independent review without unresolved behavioral or test-logic findings. ### Key changed classes `DataExecutionVisitor`, `DataRegionStateMachine`, `LoadFileNotReadyException`, `DataRegion`, `SnapshotTaker`, `LoadTsFileManager`, `TsFileWriterManager`, `LoadTsFileSnapshot`, and `LoadTsFileCleaner`, plus regression tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
