Hi all, Following up on my own proposal, since Dmitri questioned the new entity type on the issue and he had a point. I spent some time in the code, and there are two things already in Polaris that I underestimated. Here's how I'd see the options now.
What exists today 1. commitTransaction is atomic on both persistence families. The handler stages every table change and then writes all entities in one go: relational JDBC in one database transaction with version checks, NoSQL as one commit against the catalog's named pointer. 2. The NoSQL persistence is already shaped like Git: a catalog's content is an immutable state object behind a named pointer, changes are commits with a single CAS, and there's a commit log. The pointer is literally called cat/<id>/heads/main. Three ways to get to "promote many tables atomically" A. My original proposal: a CATALOG_BRANCH entity on top of per-table Iceberg branch refs. Looking at it again, the entity mostly duplicates what the refs already say, and promotion still touches every member table. I'd drop it. B. No new server concept at all. A client creates a branch ref on each table in one transactions/commit, writes to it as usual, and promotes with one more transactions/commit that, per table, asserts main is still at its base snapshot (assert-ref-snapshot-id) and moves main to the branch head (set-snapshot-ref). Either every table moves or none does, on every backend, today. What it can't do comes from Iceberg table branches: schemas are per table, so a schema change on the branch is visible on main; tables, views and namespaces can't be created or dropped "on a branch"; and the client has to know the member list. I checked that this works with a small test against the REST handler (the in-memory metastore the service tests use): three tables, branch created in one commit, written to, promoted in one commit. When main of one table was moved by another writer after branching, the promotion was rejected and none of the three tables changed. Happy to turn it into a proper test and a docs page. C. Real catalog branches as more named pointers, cat/<id>/heads/<branch>, on the NoSQL persistence. Creating a branch is one new pointer; promoting is a single CAS when main hasn't moved, for any number of tables; schema changes and created or dropped tables live on the branch; and it's the same model as Nessie. It only works on NoSQL, and it needs real work: the reference name threaded through the places that assume main (NoSqlMetaStore, References and the Table/Namespace/Policy mappings), a way for clients to pick a branch, and above all garbage collection. Today CatalogRetainedIdentifier only retains what heads/main reaches, and Iceberg snapshot expiry or orphan-file removal run on main can delete files that only a branch's table metadata references. Nessie needed a dedicated GC for this, so I'd want that settled before any code. What I'd suggest - Do B now: a documented recipe plus "polaris branch create/promote/drop" commands in the CLI. No persistence or REST changes, works on every backend, and users get atomic multi-table promotion in the next release. It also tells us which of B's limits people actually hit. - Design C here on the list, starting with the two questions that decide the rest: 1. How does a client select a branch so engines don't change? A warehouse suffix like "sales@backfill", a prefix segment from /v1/config, or a header? 2. How do we keep GC safe with several heads? And a broader one: is NoSQL the intended long-term default persistence? A NoSQL-only feature is much easier to accept if it is. I wrote a longer comparison with the details and a table, and can share it as a doc if that's useful. If B sounds reasonable, I can start on the CLI part and the docs while we work out C. Thanks, Ankur On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on catalog-level branching, tracked in > https://www.google.com/url?q=https://github.com/apache/ > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several users > have asked for it > there, and Dmitri suggested taking it to the list. Below is a first > proposal. I'd > much rather shape it with you than arrive with a finished design, so > please poke > holes in it. > > 1. The problem > > Iceberg already lets you branch a single table: a branch is just another > snapshot > ref in the table metadata. That works well for one table, but teams that > run > backfills, migrations or "write-audit-publish" pipelines across many > tables hit > three walls: > > - Creating or merging a branch across N tables takes N separate calls. > - Promotion isn't atomic. While tables are merged one by one, readers can > see > some tables with the new data and others without it. With hundreds of > tables, > that window is long, and a failure halfway through leaves the catalog in a > mixed > state someone has to repair by hand. > - There's no cheap way to ask "which tables have branch X?" without > loading every > table's metadata. > > Today the answer is either to adopt Nessie, or to script it and accept the > non-atomic window. > > 2. What I'm proposing > > A catalog branch: a named, Polaris-managed group of per-table Iceberg > branches > that is created and promoted as a unit. > > polaris branch create backfill-2026 --from main --namespace sales > ... jobs write to the "backfill-2026" branch of each table as usual ... > polaris branch promote backfill-2026 --into main # all tables, or none > > Design goals: > > - Engines don't change. Writers and readers use the standard Iceberg > branch API > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref" > field > of REST commits). Nothing new is needed in Spark, Flink, Trino or > PyIceberg. > - Build on what Polaris already has. Per-table state stays in Iceberg > metadata, > and promotion reuses the transaction workspace that commitTransaction > already > uses to apply changes to several tables as one atomic metastore update. > - Keep the first version small and safe. Fast-forward promotion only, > explicit > failure on conflict, no merge of diverged histories. > > 3. How it would work > > 3.1 A new entity > > A CATALOG_BRANCH entity, scoped to a catalog, holding: > > - the branch name, > - its scope (a namespace, optionally recursive, or an explicit list of > tables), > - for each member table: the table id and the snapshot id of main when the > branch was created (the base), > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED). > > Keeping membership in the entity answers "which tables have branch X?" > with one > read, instead of loading every table's metadata. > > 3.2 Create > > For every table in scope, add an Iceberg branch ref pointing at that > table's > current main snapshot, and write the CATALOG_BRANCH entity. All of it goes > through the transaction workspace, so either every table gets the ref and > the > entity exists, or nothing changes. > > 3.3 Write > > Nothing new. Jobs commit to the table's branch ref through the normal REST > updateTable path. Polaris only needs to reject commits that would move a > catalog branch's ref outside of Polaris' control in ways that break > promotion > (for example deleting the ref). Whether that check is worth having in v1 > is one > of the questions below. > > 3.4 Promote > > For every member table, in one transaction: > > 1. load the current metadata, > 2. check that main still points at the recorded base snapshot (fast-forward > check), > 3. set main to the branch head, > 4. optionally remove the branch ref. > > If any table fails the check, nothing is promoted, and the response lists > the > tables that moved since the branch was created. Readers either see the old > state > of every table or the new state of every table. > > 3.5 Drop > > Remove the branch refs from member tables and mark the entity DROPPED. > Snapshot > expiry then cleans up the unreferenced data as it does today. > > 3.6 API and authorization > > A Polaris REST extension under the catalog API (create, list, get, > promote, drop), > plus CLI commands. Two options for authorization: new privileges > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring > TABLE_WRITE_DATA > on every member table. The second is simpler, but slow to check for very > large > branches. > > 4. Edge cases I've thought about so far > > - Tables created in the namespace after the branch exists: not members in > v1. > They can be added explicitly. > - A member table dropped or renamed while the branch is active: promotion > fails > with a clear error. Dropping the branch still works. > - Schema changes made on the branch: carried by the branch snapshots, so > promoting them is part of the same atomic switch. > - Concurrent promotions of two branches that touch the same table: the > second one > fails the fast-forward check for that table, so nothing is promoted twice. > - Federated and static-facade catalogs: not supported, same as > commitTransaction > today. > > 5. Out of scope for a first version > > - Merging histories that have diverged (rebase or cherry-pick across > tables). > - Tags, commit logs and time travel at the catalog level, Nessie style. > - Adding this to the Iceberg REST spec. If it proves useful, I'd be glad to > propose a generalized version to the Iceberg community later. > > 6. Questions for the community > > 1. Atomicity across backends: promotion depends on a multi-entity > compare-and-set. Does every persistence backend (relational JDBC, NoSQL, > DynamoDB) give the same guarantee here, or does a backend need to advertise > support? > 2. How large should a single promotion be allowed to get? With thousands of > tables, the transaction gets big. Should there be a limit, or a two-phase > approach? > 3. Should Polaris protect a catalog branch's refs from direct deletion > through > the REST API? > 4. Is a new entity type the right home, or would you prefer a > property-based > approach on namespaces? > 5. Any appetite for aligning the vocabulary with Nessie's, given the > earlier > discussions about the two projects? > > I'm happy to present this at the next community sync and to do the > implementation in small PRs: entity and persistence first, then create and > drop, > then promote, then the CLI and docs. > > Thanks, > Ankur Kotwal >
