shivadarshan-devadiga opened a new pull request, #19027: URL: https://github.com/apache/iceberg/pull/19027
Closes #19026. `RewriteManifests.deleteManifest` checks that each explicitly deleted manifest is still in the current snapshot, but `ManifestFile` equality only compares paths. When a transaction stages an append and then replaces the staged manifest in a later `RewriteManifests`, a concurrent commit makes the transaction retry. The staged manifest keeps its path but inherits a new sequence number and first row ID from the new manifest list, so the check passes. The caller's replacement, written from the entries of the first attempt, is then committed with the old data sequence number (format v2) or with row IDs that overlap the concurrent commit (v3 and v4). This follows apache/iceberg-cpp#1104: `validateDeletedManifests` now also compares the sequence number, min sequence number and first row ID of each deleted manifest with the current manifest at the same path, and fails with a `ValidationException` if any of them changed. Committed manifests don't change these values, so only a manifest staged earlier in a retried transaction is rejected. The existing "could not be found" check and message are unchanged. The regression test `testReplaceStagedManifestAfterTransactionRetry` is from manuzhang/iceberg#214 by @manuzhang, changed to expect the validation failure and to check that the table keeps the concurrent commit. `testReplaceCommittedManifestAfterTransactionRetry` checks that replacing an already committed manifest in a transaction that retries still commits. Testing: - `TestRewriteManifests` passes (128 tests, 8 skipped). Without the change in `BaseRewriteManifests`, `testReplaceStagedManifestAfterTransactionRetry` fails for format versions 2, 3 and 4, and the rest of the suite passes. - `TestTransaction` and `TestRowLineageMetadata` pass. - `spotlessCheck` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
