shivadarshan-devadiga opened a new pull request, #19027:
URL: https://github.com/apache/iceberg/pull/19027

   Closes #19026.
   
   `RewriteManifests.deleteManifest` checks that each explicitly deleted 
manifest is still in the current snapshot, but `ManifestFile` equality only 
compares paths. When a transaction stages an append and then replaces the 
staged manifest in a later `RewriteManifests`, a concurrent commit makes the 
transaction retry. The staged manifest keeps its path but inherits a new 
sequence number and first row ID from the new manifest list, so the check 
passes. The caller's replacement, written from the entries of the first 
attempt, is then committed with the old data sequence number (format v2) or 
with row IDs that overlap the concurrent commit (v3 and v4).
   
   This follows apache/iceberg-cpp#1104: `validateDeletedManifests` now also 
compares the sequence number, min sequence number and first row ID of each 
deleted manifest with the current manifest at the same path, and fails with a 
`ValidationException` if any of them changed. Committed manifests don't change 
these values, so only a manifest staged earlier in a retried transaction is 
rejected. The existing "could not be found" check and message are unchanged.
   
   The regression test `testReplaceStagedManifestAfterTransactionRetry` is from 
manuzhang/iceberg#214 by @manuzhang, changed to expect the validation failure 
and to check that the table keeps the concurrent commit. 
`testReplaceCommittedManifestAfterTransactionRetry` checks that replacing an 
already committed manifest in a transaction that retries still commits.
   
   Testing:
   - `TestRewriteManifests` passes (128 tests, 8 skipped). Without the change 
in `BaseRewriteManifests`, `testReplaceStagedManifestAfterTransactionRetry` 
fails for format versions 2, 3 and 4, and the rest of the suite passes.
   - `TestTransaction` and `TestRowLineageMetadata` pass.
   - `spotlessCheck` passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] Core: Reject a replaced... shivadarshan-devadiga (via GitHub) via issues

Reply via email to