fishfishfishfishaa opened a new pull request, #9284: URL: https://github.com/apache/paimon/pull/9284
This PR addresses the data-loss issue reported in [#9236](https://github.com/apache/paimon/issues/9236). The scope of this PR is intentionally limited to preventing an incomplete `END_INPUT` (`Long.MAX_VALUE`) committable from being committed during recovery before global `END_INPUT` is complete. In particular, it fixes the recovery semantics that could otherwise cause later `END_INPUT` committables to be filtered out and result in data loss. This PR does **not** address the `END_INPUT` watermark semantics discussed in the same issue. Watermark handling has different checkpoint/recovery semantics and will be considered separately to keep this bug fix focused and avoid unnecessarily expanding the commit protocol changes. ### Purpose Fix a recovery data-loss issue where a partial `Long.MAX_VALUE` END_INPUT committable could be committed before global END_INPUT was complete. `Long.MAX_VALUE` alone cannot distinguish a writer that has reached END_INPUT from a globally complete END_INPUT. After a checkpoint and failover, the previous recovery path could commit such a partial committable. When the remaining END_INPUT committables later arrived, `filterAndCommit` could filter them as already committed, losing their data. This change persists a durable `completeEndInput` state. During recovery, END_INPUT committables from an incomplete state are kept pending and merged with subsequent END_INPUT committables; END_INPUT committables from a complete state retain the existing recovery-commit behavior. `endInput()` also drains buffered inputs before publishing `completeEndInput`, so a `Long.MAX_VALUE` commit is only exposed after the logical END_INPUT is complete. The change preserves normal checkpoint commit behavior and the existing `partition.mark-done.recover-from-state` behavior for complete recovered END_INPUT commits. ### Tests - `testPartialEndInputMergedBeforeOldCheckpointNotification`: restores a partial END_INPUT committable without committing it, merges the remaining END_INPUT data after recovery, and verifies that a delayed notification for the restored checkpoint commits the complete result without losing data. - `testNotTriggerPartitionMarkDownWhenRecoverFromState`: verifies that a complete recovered END_INPUT committable is committed with `partition.mark-done.recover-from-state=false`, without creating the partition `_SUCCESS` marker. - `testTriggerPartitionMarkDownWhenRecoverFromCompleteEndInputState`: verifies that a complete recovered END_INPUT committable still triggers partition mark-done recovery when the option is enabled. - `testMergeEndInputCommittables`: verifies merging restored multi-table END_INPUT committables, including committables for the same and different tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
