developer-rpai opened a new issue, #18384:
URL: https://github.com/apache/iceberg/issues/18384
## Feature Request / Improvement
Spark 4.2 introduces the DSv2 Transaction API
(`org.apache.spark.sql.connector.catalog.TransactionalCatalogPlugin`,
`org.apache.spark.sql.connector.catalog.transactions.{Transaction,
TransactionInfo}`, all `@since 4.2.0`). This sub-issue tracks implementing it
in the Iceberg Spark connector, as listed in the Spark 4.2 feature tracker
#17959.
### Proposed scope
- Implement `TransactionalCatalogPlugin` in the Spark catalog(s) so
`beginTransaction(TransactionInfo)` returns an Iceberg-backed `Transaction`.
- Wire the transaction lifecycle to Iceberg core's `Transaction` API: stage
writes under the transaction, `commit()` commits them, `abort()` discards
staged changes (idempotent), `close()` releases resources (idempotent).
- Track the read set for conflict resolution at commit time; implement
`registerScans(Scan[])` so Spark can offer materialized scans (cache reuse) for
registration against the transaction.
- Decide transaction granularity (catalog-level transaction with per-table
staged writes) and document interaction with existing write paths
(`SupportsAtomicV2Writes`, row-level operations) and Iceberg snapshot-isolation
semantics.
### Open design questions (for discussion before implementation)
1. Which SQL statements trigger transactional execution, and how do mixed
transactional/non-transactional writes behave?
2. Conflict detection/resolution at commit time vs. delegating to Iceberg's
existing snapshot-conflict checks.
3. Whether `SparkCatalog`, `SparkSessionCatalog`, or both should implement
the interface.
Part of #17959.
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
