hcrosse opened a new issue, #3743:
URL: https://github.com/apache/iceberg-rust/issues/3743

   ### Apache Iceberg Rust version
   
   `main` (`v0.11.0` and later)
   
   ### Describe the bug
   
   
[`Transaction::do_commit`](https://github.com/apache/iceberg-rust/blob/a7160a0a2318028df1a8ae88abbb3b5d611fa3d8/crates/iceberg/src/transaction/mod.rs#L221-L253)
 does not check whether a previous commit attempt already landed in the 
refreshed `TableMetadata` before reapplying actions on a retryable error (such 
as `ErrorKind::CatalogCommitConflicts`).
   
   When a catalog commit succeeds durably on the server and returns a retryable 
error (for example, a lost response on Glue, S3Tables, or S3 conditional put 
catalogs where a client retry hits a conflict), the next retry in 
[`Transaction::commit`](https://github.com/apache/iceberg-rust/blob/a7160a0a2318028df1a8ae88abbb3b5d611fa3d8/crates/iceberg/src/transaction/mod.rs#L196-L205)
 reloads the table metadata that already contains our committed snapshot and 
tries to apply the transaction actions again:
   
   - With default `FastAppendAction` (`check_duplicate = true`), the retry 
fails fatally with `ErrorKind::DataInvalid` (`Cannot add files that are already 
referenced by table`).
   - With `FastAppendAction::with_check_duplicate(false)`, the retry writes an 
orphan manifest list (`snap-<id>-1-<uuid>.avro`) and fails fatally with 
`ErrorKind::DataInvalid` (`Snapshot already exists for: '<id>'`). On `v0.11.0` 
(before snapshot IDs were reused across retries), this instead committed a 
duplicate snapshot and doubled rows returned by `Table::scan()`.
   
   ### To Reproduce
   
   1. Wrap `MemoryCatalog` so the first `update_table` call applies the commit 
durably and then returns `Error::new(ErrorKind::CatalogCommitConflicts, 
...).with_retryable(true)`.
   2. Run a `Transaction` with `FastAppendAction` (either with default 
`check_duplicate = true` or `with_check_duplicate(false)`) and call 
`tx.commit(&catalog).await`.
   3. Observe that `commit()` fails with `ErrorKind::DataInvalid` even though 
the snapshot was durably committed on the first attempt.
   
   ### Expected behavior
   
   Before reapplying actions on retry, `Transaction::do_commit` should check 
whether any snapshot `(snapshot_id, manifest_list)` submitted by a prior 
attempt of the transaction is already present in the refreshed `TableMetadata` 
(matching Iceberg Java's `checkCommitStatus`). If so, `do_commit` should return 
`Ok(refreshed)` without rewriting manifests or calling `catalog.update_table` 
again.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to