kotwal-itpro opened a new issue, #4095:
URL: https://github.com/apache/iceberg-python/issues/4095
### Feature Request / Improvement
The REST spec has an endpoint that commits changes to several tables in one
atomic operation, `POST /v1/{prefix}/transactions/commit`: either every table's
requirements hold and all of them are updated, or nothing changes. The Java
client exposes it as `RESTCatalog.commitTransaction(TableCommit...)`, and
servers such as Apache Polaris implement it. PyIceberg has no way to call it
today; every commit goes to a single table.
**Use case.** Publishing changes to several related tables at once, so
readers never see some tables updated and others not:
- write-audit-publish: stage a day's data on a branch of each table, check
it, then move every table's `main` to its branch head in one commit that
asserts `main` hasn't moved;
- backfills that rewrite a set of fact and dimension tables together.
Apache Polaris documents this pattern with the multi-table commit
(apache/polaris#5740), but from Python it currently needs hand-built REST
requests.
**Proposal.** A small API on top of the existing `Transaction`, with no
change to how single-table transactions behave:
```python
orders_tx = orders.transaction()
orders_tx.manage_snapshots().set_current_snapshot(snapshot_id=orders_branch_head)
# or any other change
items_tx = items.transaction()
items_tx.append(df)
catalog.commit_transactions(orders_tx, items_tx)
```
- `Catalog.commit_transactions(*transactions)` collects each transaction's
requirements (plus the `AssertTableUUID` that `commit_transaction` already
adds) and updates, and sends them as one `CommitTransactionRequest`.
- The REST catalog checks for the `POST /v1/{prefix}/transactions/commit`
endpoint in the config response. Catalogs that don't support it raise
`NotImplementedError` with a clear message; the base `Catalog` does the same,
so other catalogs can add support later.
- The endpoint returns 204 with no metadata, so each table is reloaded after
a successful commit, and the transactions are marked as committed.
- Errors map like `commit_table`: 409 to `CommitFailedException`,
500/502/504 to `CommitStateUnknownException`.
- No automatic retry in the first version. A 409 doesn't say which table
conflicted, and replaying all producers across tables is a bigger design; the
caller re-stages and commits again, as with the Java client. On a definite
failure, the uncommitted manifests of every transaction are cleaned up; on an
unknown outcome they are kept, the same rule `commit_transaction` follows.
- A transaction can appear only once, and two transactions on the same table
are rejected before anything is sent.
**Not in scope.** This is the existing multi-table *commit*, not multi-table
transactions with read isolation, which apache/iceberg#10617 tracks for the
spec.
I'd be happy to implement this, with unit tests against a mocked REST server
and an integration test against the REST catalog fixture. Does this API shape
work for you, or would you prefer a different entry point (for example a
context manager on the catalog)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
