hcrosse opened a new issue, #3743: URL: https://github.com/apache/iceberg-rust/issues/3743
### Apache Iceberg Rust version `main` (`v0.11.0` and later) ### Describe the bug [`Transaction::do_commit`](https://github.com/apache/iceberg-rust/blob/a7160a0a2318028df1a8ae88abbb3b5d611fa3d8/crates/iceberg/src/transaction/mod.rs#L221-L253) does not check whether a previous commit attempt already landed in the refreshed `TableMetadata` before reapplying actions on a retryable error (such as `ErrorKind::CatalogCommitConflicts`). When a catalog commit succeeds durably on the server and returns a retryable error (for example, a lost response on Glue, S3Tables, or S3 conditional put catalogs where a client retry hits a conflict), the next retry in [`Transaction::commit`](https://github.com/apache/iceberg-rust/blob/a7160a0a2318028df1a8ae88abbb3b5d611fa3d8/crates/iceberg/src/transaction/mod.rs#L196-L205) reloads the table metadata that already contains our committed snapshot and tries to apply the transaction actions again: - With default `FastAppendAction` (`check_duplicate = true`), the retry fails fatally with `ErrorKind::DataInvalid` (`Cannot add files that are already referenced by table`). - With `FastAppendAction::with_check_duplicate(false)`, the retry writes an orphan manifest list (`snap-<id>-1-<uuid>.avro`) and fails fatally with `ErrorKind::DataInvalid` (`Snapshot already exists for: '<id>'`). On `v0.11.0` (before snapshot IDs were reused across retries), this instead committed a duplicate snapshot and doubled rows returned by `Table::scan()`. ### To Reproduce 1. Wrap `MemoryCatalog` so the first `update_table` call applies the commit durably and then returns `Error::new(ErrorKind::CatalogCommitConflicts, ...).with_retryable(true)`. 2. Run a `Transaction` with `FastAppendAction` (either with default `check_duplicate = true` or `with_check_duplicate(false)`) and call `tx.commit(&catalog).await`. 3. Observe that `commit()` fails with `ErrorKind::DataInvalid` even though the snapshot was durably committed on the first attempt. ### Expected behavior Before reapplying actions on retry, `Transaction::do_commit` should check whether any snapshot `(snapshot_id, manifest_list)` submitted by a prior attempt of the transaction is already present in the refreshed `TableMetadata` (matching Iceberg Java's `checkCommitStatus`). If so, `do_commit` should return `Ok(refreshed)` without rewriting manifests or calling `catalog.update_table` again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
