Hi Qiegang, Welcome to the Polaris community!
Thanks for starting this thread. Multi-table transactions come up regularly here :) (for instance see #3469 about catalog-level branching for atomic multi-table operations, #5807 about multi-table commit not validating requirements on tables without updates, etc). So yes, I think there is interest, and starting as an optional Polaris extension is a good idea (waiting for an Iceberg proposal). For a change of this size, I would suggest: - Let's keep the design discussion here in this thread. Decisions are made on the mailing list. - I think it's great to create a GH Issue with the "proposal" label, pointing to this thread and the design doc. It will be the umbrella issue to track the work and the PRs. - If you can, share the design doc as a Google Doc open for comments, or PR with md files. - Once we have consensus on the API and the persistence SPI, we can go with small and focused PRs. I checked and your branch is around 19k lines across ~ 100 files, so I think split is welcome, else the review will be tricky. A a reminder, new public interfaces and extension points should be discussed and agreed (lazy consensus at least) on the dev mailing list. As you are moving the write path to Iceberg table branches, I would wait for the updated design before a detailed review. In the meantime, a few early thoughts from a first read :) : - the PoC relies on PostgreSQL for atomicity (row locks, the commit_scn count row, etc). Polaris has several persistence backends now (JDBC, NoSQL with MongoDB/DynamoDB/...). Especially the NoSQL ones don't rely on relational transactions. Upstream, I think the transaction state should go through a persistence SPI, with JDBC as the first implementation. It would help to describe the minimal operations that backends have to provide. - With "Universal CTS", all commits of a catalog serialize one one row, even for users who never use transactions. Maybe a feature flag for that would be appropriate. - The txn_snapshot/txn_id parameters on loadTable extend an Iceberg endpoint, and the snapshot lifecycle is an Iceberg question. We try to stay aligned with the spec on the Iceberg endpoint, so I suggest sharing the PoC on the Iceberg dev mailing list also. - I'm completely fine with autorization on the transaction endpoints not complete yet. I think the new operations have to be mapped to the Polaris privilege model, so they also work with the pluggable authorizers (OPA, Ranger, ...). - Today, COMMIT reuses the transaction workspace buffering, which PR #5088 (currently in review) is replacing with an explicit buffering. Probably a good idea to rebase your branch on main and following #5088. Also, the README says the copyright is held by Cloudera. It will probably change before any merge (and depending of what has been done, maybe a SGA would be required and an IP-Clearance). Let me know if I can help on this. Regards JB On 2026-10-19 19:30 UTC Qiegang Long wrote: > Hi all, > > Thanks to everyone who came to the multi-table transactions talk at > Lakehouse Day in Glasgow, and for the conversations afterwards. > Several of you suggested continuing here. I'm new to the Polaris > community: I built this as a proof of concept on Polaris, and this is > the first time I'm bringing it to the list. > > Iceberg already has most of the foundation for transactions engines > can use: the REST commitTransaction bundle commits several tables > atomically, and table branches can stage writes. What's missing is the > engine usable transaction around them: BEGIN ... COMMIT across > statements, with one consistent read point across tables. The > discussion of this on the Iceberg list has been quiet for a while, and > I built the PoC to restart it with something concrete. > > The PoC runs Spark and Trino against a Polaris fork: multi-statement > transactions at READ COMMITTED and REPEATABLE READ, a read point > shared across engines, and fan-out writes from independent workers > that commit atomically. After conversations with Iceberg folks at the > conference, I'm moving the write path onto Iceberg table branches, so > writes become standard REST updates; I'll follow up in this thread > with the updated code. > > For example, in Spark with the Polaris plugin: > > BEGIN TRANSACTION; > INSERT INTO customers VALUES (22, 'joe', '[email protected]'); > INSERT INTO orders VALUES (2002, 22, 9.99, 'pending'); > COMMIT; -- both tables become visible together, or neither does > > Readers in Spark or Trino, at either isolation level, never see the > order without its customer. > > The change to the Iceberg REST surface is small: one optional > loadTable parameter (the read point). Everything else is Polaris-only > endpoints (minting a read point, fan-out coordination). So it could > live in Polaris first as an opt-in extension, with an Iceberg proposal > later if the community wants one. > > Would Polaris be interested in carrying it as an extension first? And > since I'm new here, I'd appreciate guidance on the usual path for a > change of this size: for example a design review here first, an issue > to track it, time at a community sync, or going straight to small PRs. > I'm happy to work whichever way suits the project. It's an early PoC > with known gaps (listed in the design doc), and JDBC/PostgreSQL only > for now. > > Talk, slides, demo video, design doc and code: > https://github.com/qlong/lakehouse-transactions > > Thanks, > Qiegang >
