Hi all, I've opened https://github.com/apache/polaris/issues/5833 as the umbrella issue to track this work, linking this thread.
I'm now updating the design doc for the branch-based write path. I'll keep the first version focused on the core: the transaction semantics and the REST API. The persistence SPI and the other points JB raised (the privilege-model mapping, the per-catalog setting, fitting in after #5088) I'll take up once those are settled. I'll share the doc here when it's ready, and post on the Iceberg dev list at the same time. Thanks, Qiegang On Tue, Oct 20, 2026 at 11:38 AM Qiegang Long <[email protected]> wrote: > > Hi JB, > > Thanks for the welcome and the detailed guidance, it helps a lot. > > I'll follow the process you described: open an umbrella issue with the > "proposal" label, > pointing to this thread and the design doc, and share the design doc > for comments. Agreed on > waiting for the branch-based design before a detailed review. > > Your technical points make sense, I'll address them in the updated design: > - a persistence SPI for the transaction state, with JDBC as the first > implementation, and the > minimal operations a backend must provide; > - a setting for the commit-order bookkeeping, so users who don't use > transactions don't pay for > it (per catalog, since a read point needs every commit in that > catalog to be ordered); > - mapping the new operations to the Polaris privilege model, so they > work with the pluggable > authorizers; > - rebasing on main and following #5088. > > I'll also share the PoC on the Iceberg dev list, for the loadTable > parameters and the snapshot > lifecycle. > > On copyright: I've started the conversation with Cloudera's > open-source team and will come back > to you. One question that would help: if the code comes in as new, > focused PRs against current > main, does that follow the normal contribution path, or would you > still expect an SGA for the > PoC code? And thanks for offering to help. > > Best, > Qiegang > > On Tue, Oct 20, 2026 at 2:34 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > > Hi Qiegang, > > > > Welcome to the Polaris community! > > > > Thanks for starting this thread. > > > > Multi-table transactions come up regularly here :) (for instance see #3469 > > about catalog-level branching for atomic multi-table operations, #5807 > > about multi-table commit not validating requirements on tables without > > updates, etc). > > So yes, I think there is interest, and starting as an optional Polaris > > extension is a good idea (waiting for an Iceberg proposal). > > > > For a change of this size, I would suggest: > > - Let's keep the design discussion here in this thread. Decisions are made > > on the mailing list. > > - I think it's great to create a GH Issue with the "proposal" label, > > pointing to this thread and the design doc. It will be the umbrella issue > > to track the work and the PRs. > > - If you can, share the design doc as a Google Doc open for comments, or PR > > with md files. > > - Once we have consensus on the API and the persistence SPI, we can go with > > small and focused PRs. I checked and your branch is around 19k lines across > > ~ 100 files, so I think split is welcome, else the review will be tricky. A > > a reminder, new public interfaces and extension points should be discussed > > and agreed (lazy consensus at least) on the dev mailing list. > > > > As you are moving the write path to Iceberg table branches, I would wait > > for the updated design before a detailed review. > > > > In the meantime, a few early thoughts from a first read :) : > > - the PoC relies on PostgreSQL for atomicity (row locks, the commit_scn > > count row, etc). Polaris has several persistence backends now (JDBC, NoSQL > > with MongoDB/DynamoDB/...). Especially the NoSQL ones don't rely on > > relational transactions. Upstream, I think the transaction state should go > > through a persistence SPI, with JDBC as the first implementation. It would > > help to describe the minimal operations that backends have to provide. > > - With "Universal CTS", all commits of a catalog serialize one one row, > > even for users who never use transactions. Maybe a feature flag for that > > would be appropriate. > > - The txn_snapshot/txn_id parameters on loadTable extend an Iceberg > > endpoint, and the snapshot lifecycle is an Iceberg question. We try to stay > > aligned with the spec on the Iceberg endpoint, so I suggest sharing the PoC > > on the Iceberg dev mailing list also. > > - I'm completely fine with autorization on the transaction endpoints not > > complete yet. I think the new operations have to be mapped to the Polaris > > privilege model, so they also work with the pluggable authorizers (OPA, > > Ranger, ...). > > - Today, COMMIT reuses the transaction workspace buffering, which PR #5088 > > (currently in review) is replacing with an explicit buffering. Probably a > > good idea to rebase your branch on main and following #5088. > > > > Also, the README says the copyright is held by Cloudera. It will probably > > change before any merge (and depending of what has been done, maybe a SGA > > would be required and an IP-Clearance). Let me know if I can help on this. > > > > Regards > > JB > > > > On 2026-10-19 19:30 UTC Qiegang Long wrote: > > > Hi all, > > > > > > Thanks to everyone who came to the multi-table transactions talk at > > > Lakehouse Day in Glasgow, and for the conversations afterwards. > > > Several of you suggested continuing here. I'm new to the Polaris > > > community: I built this as a proof of concept on Polaris, and this is > > > the first time I'm bringing it to the list. > > > > > > Iceberg already has most of the foundation for transactions engines > > > can use: the REST commitTransaction bundle commits several tables > > > atomically, and table branches can stage writes. What's missing is the > > > engine usable transaction around them: BEGIN ... COMMIT across > > > statements, with one consistent read point across tables. The > > > discussion of this on the Iceberg list has been quiet for a while, and > > > I built the PoC to restart it with something concrete. > > > > > > The PoC runs Spark and Trino against a Polaris fork: multi-statement > > > transactions at READ COMMITTED and REPEATABLE READ, a read point > > > shared across engines, and fan-out writes from independent workers > > > that commit atomically. After conversations with Iceberg folks at the > > > conference, I'm moving the write path onto Iceberg table branches, so > > > writes become standard REST updates; I'll follow up in this thread > > > with the updated code. > > > > > > For example, in Spark with the Polaris plugin: > > > > > > BEGIN TRANSACTION; > > > INSERT INTO customers VALUES (22, 'joe', '[email protected]'); > > > INSERT INTO orders VALUES (2002, 22, 9.99, 'pending'); > > > COMMIT; -- both tables become visible together, or neither does > > > > > > Readers in Spark or Trino, at either isolation level, never see the > > > order without its customer. > > > > > > The change to the Iceberg REST surface is small: one optional > > > loadTable parameter (the read point). Everything else is Polaris-only > > > endpoints (minting a read point, fan-out coordination). So it could > > > live in Polaris first as an opt-in extension, with an Iceberg proposal > > > later if the community wants one. > > > > > > Would Polaris be interested in carrying it as an extension first? And > > > since I'm new here, I'd appreciate guidance on the usual path for a > > > change of this size: for example a design review here first, an issue > > > to track it, time at a community sync, or going straight to small PRs. > > > I'm happy to work whichever way suits the project. It's an early PoC > > > with known gaps (listed in the design doc), and JDBC/PostgreSQL only > > > for now. > > > > > > Talk, slides, demo video, design doc and code: > > > https://github.com/qlong/lakehouse-transactions > > > > > > Thanks, > > > Qiegang > > >
