Caideyipi opened a new pull request, #19241:
URL: https://github.com/apache/iotdb/pull/19241

   ## Description
   
   A replicated LOAD PREPARE or COMMIT can arrive before a missing PIECE is 
retried after reconnect. Returning a permanent LOAD error lets consensus retire 
the command before the replica has finished the task. This draft retries only 
temporarily incomplete replicated commands and preserves completed task 
identities across retries and restarts.
   
   **This is an incomplete fix and is not ready to merge.** The remaining 
recovery requirements below belong to the same LOAD retry path. Some underlying 
behaviors predate this patch, but the retry guarantee still depends on 
addressing them.
   
   ### Changes currently implemented
   
   - Introduce `LoadFileNotReadyException` and map it to `WRITE_PROCESS_REJECT` 
for replicated PREPARE/COMMIT only. Permanent failures and local command 
failures retain their existing error classification.
   - Persist COMMIT/ABORT outcomes outside task staging, include them in LOAD 
snapshots, and use the bounded in-memory history as a cache. Record empty ABORT 
outcomes and ignore late replicated PIECEs for completed tasks.
   - Coordinate staging snapshot copies with COMMIT/ABORT. Migrate legacy 
terminal records before retention listener registration, preserve staging when 
migration fails, and fence lazy manager initialization against region deletion.
   - Correct the lazy-initialization lock order relative to snapshot flushing.
   
   ### TODO before marking ready, and how to adapt
   
   1. [ ] **Preserve acknowledged, unfinished staging.** Adapt 
`LoadTsFileManager.stop`, `TsFileWriterManager.close`, and 
`LoadTsFileCleaner.reclaimIfIdle` to distinguish coordinator temporary files 
from consensus-owned accepted state. Ordinary shutdown must close handles while 
preserving recoverable payload and progress. Idle time alone must not delete 
accepted replicated data: remote requests use a negative local search index, so 
the existing watermark guard does not protect them. Reclamation needs an 
explicit terminal decision, a proven safe replay frontier, or an implemented 
payload reconstruction protocol. Otherwise, a sender that already retired the 
PIECE batch can only retry PREPARE/COMMIT against missing data indefinitely.
   
   2. [ ] **Make COMMIT import recoverable and idempotent.** Adapt 
`LoadTsFileManager.loadAll`, `TsFileWriterManager.loadAll`, and the 
`DataRegion` import path. Persist an import plan/COMMITTING state with stable 
per-task, per-partition installation identities and enough information to 
recognize published files after restart. Recovery must finish missing 
installations without reinstalling completed ones or repeating their 
publication effects. The current sequence imports files before persisting 
COMMIT, while ordinary import allocates a new version/name each time. Keeping 
the staged source does not by itself make replay safe. Moving the final COMMIT 
record before import would leave a different missing-data window and is 
insufficient.
   
   3. [ ] **Coordinate the terminal decision with replication WAL durability.** 
Adapt the terminal-hit paths and WAL logging in 
`LoadTsFileManager.loadAll`/`deleteAll`. Currently the terminal record can 
become durable before the command reaches WAL, and a subsequent terminal-hit 
return skips WAL logging. Define a replayable decision/intent and local 
application-completion protocol so successful retries also satisfy outstanding 
replication obligations. An in-memory dispatcher re-offer is insufficient 
across restart or queue overflow. The design must reconcile stable decision 
identity, WAL replay, and application progress; a local terminal file alone 
must not authorize success when durable replication information is missing.
   
   4. [ ] **Apply one state/ownership model to snapshots and lifecycle 
operations.** A bounded direction is a region-local task protocol such as 
RECEIVING -> PREPARED -> COMMITTING -> COMMITTED, with explicit permitted 
ABORTED transitions. Define the durability and acknowledgement conditions for 
every transition before implementation. Capture data files, active task/import 
state, terminal outcomes, and replay position from a consistent boundary. 
Preserve the existing staging-copy boundary, migration-before-listener 
ordering, and consistent lock order. Separate ordinary stop from region 
deletion. Terminal-history garbage collection needs a proven replay frontier; 
arbitrary TTL or an in-memory receive index is insufficient. Measure 
terminal-record growth and staging/snapshot copy latency. This does not require 
adding cross-region atomic LOAD or rollback semantics.
   
   5. [ ] **Validate the recovery boundaries above.** Add failure-oriented 
coverage for acknowledged PIECE -> normal stop/restart -> later PREPARE/COMMIT; 
acknowledged PIECE -> idle reclamation attempt -> later command; partial/full 
import -> process termination before terminal publication -> restart/retry; 
terminal publication -> WAL failure/restart or dispatch queue overflow -> 
replica catch-up; and snapshots spanning these transitions. Assert 
installed-file identity/count, relevant publication effects, recovered task 
state, and replica catch-up rather than only successful status codes. Re-run 
the focused tests and the restart ClusterIT on the completed implementation, 
then perform a complete independent review before marking ready.
   
   ### Validation and revision limits
   
   - Current candidate: `a7d199b65162c9d62bad1c4e64883e33472fc58f`.
   - Standard focused local Maven reactor: **80 tests passed**, with zero 
failures, errors, or skips. Coverage includes state-machine classification, 
LOAD manager/snapshot/cleaner behavior, selected DataRegion lifecycle tests, 
and selected IoTDB snapshot tests. All 14 test methods added in this diff were 
run locally.
   - Spotless, Checkstyle, forbidden-API checks, and `git diff --check` passed.
   - These results do not establish crash safety for the unfinished TODOs 
above. No current-candidate ClusterIT, full CI, Chinese-locale compilation, 
process-kill crash test, or performance benchmark has been run.
   - The candidate was developed and tested on recorded baseline 
`6e7448d9bebd644259b695dfc8774d3d9f91252e`. The refreshed target `master` was 
`54bff3d9a421f9ba148fda3a26626df570422f54` when this draft was prepared. The 
candidate was not rebased, and an integration merge with that newer target has 
not been validated.
   - Review findings remain outstanding at the design level; this draft has not 
passed an issue-free independent review.
   
   This PR has:
   
   - [x] Added regression unit tests and run every newly added test locally.
   - [ ] Completed the recovery protocol TODOs above.
   - [ ] Validated the completed candidate with the restart ClusterIT.
   - [ ] Passed a complete independent review without unresolved behavioral or 
test-logic findings.
   
   ### Key changed classes
   
   `DataExecutionVisitor`, `DataRegionStateMachine`, 
`LoadFileNotReadyException`, `DataRegion`, `SnapshotTaker`, 
`LoadTsFileManager`, `TsFileWriterManager`, `LoadTsFileSnapshot`, and 
`LoadTsFileCleaner`, plus regression tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to