Hi all,
Thanks to everyone who came to the multi-table transactions talk at
Lakehouse Day in Glasgow, and for the conversations afterwards.
Several of you suggested continuing here. I'm new to the Polaris
community: I built this as a proof of concept on Polaris, and this is
the first time I'm bringing it to the list.
Iceberg already has most of the foundation for transactions engines
can use: the REST commitTransaction bundle commits several tables
atomically, and table branches can stage writes. What's missing is the
engine usable transaction around them: BEGIN ... COMMIT across
statements, with one consistent read point across tables. The
discussion of this on the Iceberg list has been quiet for a while, and
I built the PoC to restart it with something concrete.
The PoC runs Spark and Trino against a Polaris fork: multi-statement
transactions at READ COMMITTED and REPEATABLE READ, a read point
shared across engines, and fan-out writes from independent workers
that commit atomically. After conversations with Iceberg folks at the
conference, I'm moving the write path onto Iceberg table branches, so
writes become standard REST updates; I'll follow up in this thread
with the updated code.
For example, in Spark with the Polaris plugin:
BEGIN TRANSACTION;
INSERT INTO customers VALUES (22, 'joe', '[email protected]');
INSERT INTO orders VALUES (2002, 22, 9.99, 'pending');
COMMIT; -- both tables become visible together, or neither does
Readers in Spark or Trino, at either isolation level, never see the
order without its customer.
The change to the Iceberg REST surface is small: one optional
loadTable parameter (the read point). Everything else is Polaris-only
endpoints (minting a read point, fan-out coordination). So it could
live in Polaris first as an opt-in extension, with an Iceberg proposal
later if the community wants one.
Would Polaris be interested in carrying it as an extension first? And
since I'm new here, I'd appreciate guidance on the usual path for a
change of this size: for example a design review here first, an issue
to track it, time at a community sync, or going straight to small PRs.
I'm happy to work whichever way suits the project. It's an early PoC
with known gaps (listed in the design doc), and JDBC/PostgreSQL only
for now.
Talk, slides, demo video, design doc and code:
https://github.com/qlong/lakehouse-transactions
Thanks,
Qiegang
