corgy-w commented on PR #12440: URL: https://github.com/apache/seatunnel/pull/12440#issuecomment-6788840756
@DanielLeens thanks again for the detailed review. I addressed the remaining items in `3656e78b3a`, rebased on current `dev` (`f5374addb3`) with the docs conflicts resolved against the rewritten EN/ZH pages. **Issue 1 (High): orphaned instants are now reclaimed by the committer on restore.** `restoreCommit()` now calls `reclaimOrphanedInstants()` before committing the restored instants: it reloads the active timeline and rolls back every `commit` / `deltacommit` instant that the restored checkpoint state does not reference. This is Option A, with two deliberate boundaries: * Only commit and delta-commit actions are considered. A pending compaction or another Hudi table-service action is never touched. * Nothing is reclaimed when the restored state is empty, because there is no reference point that separates the instants of a failed attempt from those of another writer. That case (a job that never completed a checkpoint, or a pipeline started from scratch) is documented as manual cleanup, and the single-writer requirement is stated explicitly. This also covers failure shape (b) from your review: the instant of a checkpoint that handed its records over but never completed is younger than the newest restored instant, so an "older than the newest restored instant" guard would have left exactly that case behind. The safety argument is the writer-start ordering instead: the committer restores before the task group starts, and writers only create an instant on the first write, so no writer of this job can own an instant at that point. **Issue 2 (Medium): tests added.** * `shouldRollbackTheInstantWhenTheWriterClosesWithoutCheckpoint` writes a batch, closes the writer without `prepareCommit` and asserts the instant is gone from the timeline. * `shouldFailTheCheckpointWhenHudiRejectsARecord` injects an errored `WriteStatus`, asserts `HudiConnectorException` with `HUDI-07`, and asserts the rejected checkpoint leaves no instant behind. * `shouldReclaimTheInstantOfACheckpointThatNeverCompleted` writes two checkpoints, restores only the first and asserts the second instant is rolled back while the first is committed. **Issue 5 (Low): docs.** Both languages now describe the three cleanup points (writer close, `abort`, restore reclaim), the two remaining manual cases, and the single-writer assumption. The stale "heartbeat expired before the commit" bullet is replaced by "an operator or another writer rolled it back". **Issues 3 and 4** are tracked in #13224 (stats-only commit state + fault-injection E2E) and linked from the PR description. Local verification (JDK 11): `./mvnw -pl seatunnel-connectors-v2/connector-hudi -DskipIT=true test` -> 30 tests, 0 failures; `spotless:apply` clean. The branch is now mergeable again (`MERGEABLE`), and CI is running on the new head. Could you take another look when you have a moment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
