fishfishfishfishaa commented on PR #8671:
URL: https://github.com/apache/paimon/pull/8671#issuecomment-5367720793
Hi, I’ve pushed a new revision of the code. The related logic and design
have also been updated in the PR description.
In addition, below is a summary of the failover scenarios I considered and
how they are covered:
1. All writers finish before the same checkpoint
all writers EndInput
→ checkpoint C captures MAX committables
→ notifyCheckpointComplete(C)
→ final commit
This is the normal final-commit path. On subtask or global recovery:
- A MAX produced after the restore checkpoint is discarded with the failed
execution.
- A MAX contained in the restore checkpoint is restored as pending state.
- If not every writer has a restored/final MAX, no final commit occurs.
- If the final commit may have succeeded before failure, recovery may replay
it through filterAndCommit.
2. Writers finish in different checkpoints
W0 EndInput → checkpoint C1
W1 still running
W1 EndInput → checkpoint C2
notifyCheckpointComplete(C2) → final commit
C1 can commit normal data, but cannot commit the final state because W1
has not finished. A restored MAX from an early-finished writer remains pending
until a later checkpoint covers every writer’s EndInput.
3. An older checkpoint completes after EndInput arrives
checkpoint C1 snapshots
→ all writers EndInput
→ notifyCheckpointComplete(C1)
C1 must not commit MAX: its snapshot predates EndInput. Only a later
checkpoint that actually captures the final state can enable the final commit.
This remains true after either subtask or global recovery; coordinator event
arrival order is never sufficient proof of checkpoint coverage.
4. The covering checkpoint is aborted
all writers EndInput
→ checkpoint C captures MAX
→ C is aborted
No final commit occurs. The final state remains pending and must be covered
by a later successfully completed checkpoint.
5. Replayed or duplicate EndInput
A finished writer may restore and resend its Long.MAX_VALUE committable
even after the final commit succeeded.
The coordinator treats it as one terminal entry per writer, rather than
merging it as additional data. Any recovered final commit goes through
filterAndCommit, preventing duplicate effective table commits.
This also applies to empty final committables: an empty MAX still proves
that the writer has finished and must participate in global EndInput
completeness.
6. Coordinator close
close() must not discard an already accepted checkpoint-completion action.
notifyCheckpointComplete(C) queued
→ close()
→ drain queued coordinator actions
→ shutdown executor
→ close committer
Therefore, a final commit that is already queued or running completes
before the committer is closed. close() does not create an additional EndInput
commit path; it only preserves the result of the checkpoint-driven path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
