ajanko-shopify opened a new pull request, #18916: URL: https://github.com/apache/iceberg/pull/18916
This PR supersedes #17901 by @fmorillo7694. It keeps their original commit, rebases it onto `main`, and addresses the open review comment. All credit for the fix and the regression test goes to them. If they'd rather continue in #17901, I'm happy to close this PR. ## Problem `IcebergCommitter` reads `getMaxCommittedCheckpointId()` once, before the commit. Another commit for the same checkpoint can land after that read, and the second commit then applies cleanly on top of the first. The checkpoint's files are committed twice. See #17901 and #14425 for the original analysis and the deterministic reproduction. We also hit this in production with blue/green deployments, where the old job keeps running for a short time after the savepoint is taken. The old job commits checkpoint N. Seconds later, the new job commits the same committable, which it restored from the savepoint, under the same job ID. In one table, we found 10 data files that were each added twice, in consecutive snapshots. ## Changes The first commit is the original change from #17901, rebased onto `main`. The second commit makes these changes: - **Moves `MaxCommittedCheckpointIdValidator` into `SinkUtil`**, as @mxm asked in https://github.com/apache/iceberg/pull/17901#discussion_r4155439525. Its mismatch exception moves with it. - **Adapts the validator to #18303.** `DynamicCommitter` now identifies its own snapshots by sink ID, and `IcebergCommitter` still uses the job ID and operator ID. So each caller now passes a predicate that matches its own snapshot summaries. - **In `IcebergCommitter`, validates only after a restore.** This matches the up-front check since #18123. After a stateless start, a job can reuse its job ID while its checkpoint IDs start from 1 again. Without this condition, the validator would skip those commits and silently drop data. `testCommitTxnAfterStatelessRestart` caught this. Duplicate commits come from restored committables, so restored jobs, including blue/green deployments, are still protected. - **Updates the regression test** for the new `IcebergCommitter` constructor. Like #17901, this targets Flink 2.3 only. Backports can follow once the approach is approved. ## Testing - `TestIcebergCommitterDuplicateCommit` passes. It fails when the validator is disabled, with 2 snapshots instead of 1. - `TestIcebergCommitter` (336), `TestIcebergFilesCommitter` (96), `TestIcebergSink` (312), `TestDynamicCommitter` (14) and `TestDynamicIcebergSink` (50) all pass. - `spotlessCheck`, `checkstyleMain` and `checkstyleTest` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
