corgy-w commented on PR #12440:
URL: https://github.com/apache/seatunnel/pull/12440#issuecomment-6788840756

   @DanielLeens thanks again for the detailed review. I addressed the remaining 
items in `3656e78b3a`, rebased on current `dev` (`f5374addb3`) with the docs 
conflicts resolved against the rewritten EN/ZH pages.
   
   **Issue 1 (High): orphaned instants are now reclaimed by the committer on 
restore.**
   `restoreCommit()` now calls `reclaimOrphanedInstants()` before committing 
the restored instants: it reloads the active timeline and rolls back every 
`commit` / `deltacommit` instant that the restored checkpoint state does not 
reference. This is Option A, with two deliberate boundaries:
   
   * Only commit and delta-commit actions are considered. A pending compaction 
or another Hudi table-service action is never touched.
   * Nothing is reclaimed when the restored state is empty, because there is no 
reference point that separates the instants of a failed attempt from those of 
another writer. That case (a job that never completed a checkpoint, or a 
pipeline started from scratch) is documented as manual cleanup, and the 
single-writer requirement is stated explicitly.
   
   This also covers failure shape (b) from your review: the instant of a 
checkpoint that handed its records over but never completed is younger than the 
newest restored instant, so an "older than the newest restored instant" guard 
would have left exactly that case behind. The safety argument is the 
writer-start ordering instead: the committer restores before the task group 
starts, and writers only create an instant on the first write, so no writer of 
this job can own an instant at that point.
   
   **Issue 2 (Medium): tests added.**
   * `shouldRollbackTheInstantWhenTheWriterClosesWithoutCheckpoint` writes a 
batch, closes the writer without `prepareCommit` and asserts the instant is 
gone from the timeline.
   * `shouldFailTheCheckpointWhenHudiRejectsARecord` injects an errored 
`WriteStatus`, asserts `HudiConnectorException` with `HUDI-07`, and asserts the 
rejected checkpoint leaves no instant behind.
   * `shouldReclaimTheInstantOfACheckpointThatNeverCompleted` writes two 
checkpoints, restores only the first and asserts the second instant is rolled 
back while the first is committed.
   
   **Issue 5 (Low): docs.** Both languages now describe the three cleanup 
points (writer close, `abort`, restore reclaim), the two remaining manual 
cases, and the single-writer assumption. The stale "heartbeat expired before 
the commit" bullet is replaced by "an operator or another writer rolled it 
back".
   
   **Issues 3 and 4** are tracked in #13224 (stats-only commit state + 
fault-injection E2E) and linked from the PR description.
   
   Local verification (JDK 11): `./mvnw -pl 
seatunnel-connectors-v2/connector-hudi -DskipIT=true test` -> 30 tests, 0 
failures; `spotless:apply` clean. The branch is now mergeable again 
(`MERGEABLE`), and CI is running on the new head. Could you take another look 
when you have a moment?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to