NoahKusaba opened a new pull request, #3372: URL: https://github.com/apache/iceberg-rust/pull/3372
## Which issue does this PR close? - Related to #3134 (covers its idempotent-write case without making `TransactionAction` public). ## What changes are included in this PR? An engine that reruns a task can commit the same append twice. Checking for an earlier commit before `Transaction::commit` costs an extra table load and still races, because `do_commit` reloads the table and rebuilds the actions. - `FastAppendAction::with_idempotency_key(key)` records `key` in the snapshot summary (`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`). On every commit attempt it walks `main`'s ancestry in the refreshed table and skips the append if a snapshot already has the key. A racing loser fails its ref requirement, retries, finds the key and skips. A rerun appending the same files is skipped rather than rejected as a duplicate. Flink's committer uses the same pattern with `flink.job-id`. - `snapshot_with_idempotency_key(&table, key)` returns the snapshot on `main` holding a key. A caller that records a value unique to its commit call in the snapshot properties can compare it to tell whether its append was committed or skipped. - `do_commit` no longer calls `update_table` when no action produced updates or requirements, as in Java. Before, such a commit still wrote a new metadata file (for example, `expire_snapshots` with nothing to expire). An append with no files and a key commits a snapshot recording the key, so its rerun is skipped too. Keys on expired or rolled-back snapshots are not found. ## Are these changes tested? Yes, unit tests in `transaction/append.rs`: - reruns from a stale table, with the same files, and with the key below later snapshots - an append without files - a race, replayed through a mock catalog in sequence (load, rejected update, load) - the lookup helper And in `transaction/mod.rs`, a commit without updates writing no metadata file. ## AI Disclosure Drafted with Claude Code; I reviewed it and ran the tests, clippy and fmt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
