yadavay-amzn opened a new pull request, #17204: URL: https://github.com/apache/iceberg/pull/17204
Closes #17202. ## Problem When `rewrite_data_files` runs with partial progress enabled and a concurrent `expire_snapshots` removes the rewrite's starting snapshot before the file-group commits complete, the action either throws a `NullPointerException` or silently completes with zero committed rewrites - losing all compaction progress with no error surfaced to the caller. - `RewriteDataFilesCommitManager#commitFileGroups` calls `table.snapshot(startingSnapshotId).sequenceNumber()`, which throws `NullPointerException` because `Table#snapshot(long)` returns `null` for an expired snapshot (`use-starting-sequence-number` defaults to `true`). - `BaseCommitService#commitReadyCommitGroups` catches and ignores the failure via a catch-all, so it is counted as a tolerable partial-progress failure and only a `LOG.warn` is emitted. ## Change - Null-guard the starting snapshot in `commitFileGroups` with a clear message indicating the starting snapshot has been expired (likely by a concurrent `expire_snapshots`). - Propagate the non-retriable `IllegalStateException` from `BaseCommitService` instead of swallowing it, while still tolerating genuine retriable commit conflicts (`ValidationException` / `CommitFailedException`) as before. ## Testing Added `TestRewriteDataFilesCommitManager` with two tests that reproduce both failure paths (the NPE and the silent swallow). Both fail on the unpatched code and pass with the fix. Existing `TestCommitService` behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
