ifndef-SleePy commented on PR #8671: URL: https://github.com/apache/paimon/pull/8671#issuecomment-5571393633
Sorry for late response. I have some questions: #### 1. Can recovery discard checkpointed terminal committables? Could a repeated `endInput()` after recovery discard terminal committables already saved in the restored checkpoint? For example: 1. Writer W0 reaches `endInput()` and reports terminal batch A. W1 is still processing input. 2. Checkpoint C completes and includes A in W0’s state. The coordinator keeps A pending because W1 has not reached end-of-input. 3. The job recovers from C. W0 restores A, but A’s input records are already covered by C and are not replayed. 4. If W0 receives `endInput()` again, `prepareCommit()` may produce an empty batch B. The writer replaces its pending `MAX` entry with B, and the coordinator also replaces A with B. This differs from recovering to a checkpoint **before** A was produced, where replay generates a replacement batch and discarding A is correct. Here, A belongs to the restored checkpoint and has not been committed. If this sequence is possible, replacing A would lose pending data. #### 2. Failures during shutdown may not reach the caller Draining pending work addresses normal shutdown, but failures in the final commit/tag work may still leave the job or stop-with-savepoint operation reporting success. This is the remaining limitation discussed in [#9309](https://github.com/apache/paimon/pull/9309#discussion_r3827977344). The disabled non-drain stop-with-savepoint test checks normal tag creation without injecting failures. Passing that test would not establish that final commit/tag failures are propagated. #### 3. The coordinator path does not apply `end-input.watermark` The classic committer applies `end-input.watermark` when input ends, but the coordinator path only uses the reported writer watermarks. When this option is configured, the terminal commit can therefore use a different watermark depending on the commit path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
