Hi Ankur, Thanks again for taking this feature discussion to the dev ML!
Re: Option B: atomic multi-entity changes have some issues with the current Persistence framework. Basic operations work well, but there are concerns with finer points. There is a rather long discussion about that in [1]. I need some time to consider this idea. Let's see what other people think. If you could contribute a test + doc PR to Polaris for this idea, it would be great! Re: Option C: Catalog-level branches are indeed natural in the NoSQL Persistence. I'd personally prefer to explore this option from the start (but I do not insist). If there's interest in supporting the NoSQL model over JDBC, I believe it is quite possible to implement (despite the NoSQL name). That will open native catalog branches to RDBMS-based environments too. The only operational drawback would be having to run NoSQL repository maintenance [2] even on RDBMS, but I think it is quite an acceptable inconvenience for the power of the feature as a whole. The traditional JDBC persistence and the proposed "Atomic DynamoDB" [5615] persistence will remain as the "classical" options, of course. Users interested in catalog-level branches will have the option to use NoSQL with MongoDB (current codebase), DynamoDB [5725] or JDBC (future PR). I certainly consider the NoSQL Persistence framework to be a long-term feature. Specific backend support depends on user interest, I think, but the range of possibilities virtually covers the majority of existing databases. I'd also add that onboarding a new backend is rather simple. All the hard tasks are handled in common code. Re: Question 1: both the "prefix" and the "header" approaches can work, I think. Even at the same time. Re: Question 2 (GC): IMHO, GC becomes a catalog-side concern, and that is a good thing. Polaris already has some relevant concepts, e.g. OrphanFileRemovalPolicyContent [1261]. IIRC, there were some prior discussions about an external "table maintenance service" that could enforce those policies (I could not find a specific dev thread, though). [1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp [2] https://polaris.apache.org/releases/1.8.0/metastores/nosql-mongodb/#regular-maintenance [1261] https://github.com/apache/polaris/pull/1261 [5615] https://github.com/apache/polaris/issues/5615 [5725] https://github.com/apache/polaris/pull/5725 Cheers, Dmitri. On Wed, Oct 7, 2026 at 2:43 PM Ankur Kotwal <[email protected]> wrote: > Hi all, > > Following up on my own proposal, since Dmitri questioned the new entity > type on > the issue and he had a point. I spent some time in the code, and there are > two > things already in Polaris that I underestimated. Here's how I'd see the > options > now. > > What exists today > > 1. commitTransaction is atomic on both persistence families. The handler > stages > every table change and then writes all entities in one go: relational JDBC > in > one database transaction with version checks, NoSQL as one commit against > the > catalog's named pointer. > > 2. The NoSQL persistence is already shaped like Git: a catalog's content is > an > immutable state object behind a named pointer, changes are commits with a > single CAS, and there's a commit log. The pointer is literally called > cat/<id>/heads/main. > > Three ways to get to "promote many tables atomically" > > A. My original proposal: a CATALOG_BRANCH entity on top of per-table > Iceberg > branch refs. Looking at it again, the entity mostly duplicates what the > refs > already say, and promotion still touches every member table. I'd drop it. > > B. No new server concept at all. A client creates a branch ref on each > table in > one transactions/commit, writes to it as usual, and promotes with one more > transactions/commit that, per table, asserts main is still at its base > snapshot (assert-ref-snapshot-id) and moves main to the branch head > (set-snapshot-ref). Either every table moves or none does, on every > backend, > today. What it can't do comes from Iceberg table branches: schemas are per > table, so a schema change on the branch is visible on main; tables, views > and > namespaces can't be created or dropped "on a branch"; and the client has to > know the member list. > > I checked that this works with a small test against the REST handler > (the in-memory metastore the service tests use): three tables, branch > created in one commit, written to, promoted in one commit. When main of one > table was moved by another writer after branching, the promotion was > rejected and none of the three tables changed. Happy to turn it into a > proper test and a docs page. > > C. Real catalog branches as more named pointers, cat/<id>/heads/<branch>, > on the > NoSQL persistence. Creating a branch is one new pointer; promoting is a > single > CAS when main hasn't moved, for any number of tables; schema changes and > created or dropped tables live on the branch; and it's the same model as > Nessie. It only works on NoSQL, and it needs real work: the reference name > threaded through the places that assume main (NoSqlMetaStore, References > and > the Table/Namespace/Policy mappings), a way for clients to pick a branch, > and > above all garbage collection. Today CatalogRetainedIdentifier only retains > what heads/main reaches, and Iceberg snapshot expiry or orphan-file removal > run on main can delete files that only a branch's table metadata > references. > Nessie needed a dedicated GC for this, so I'd want that settled before any > code. > > What I'd suggest > > - Do B now: a documented recipe plus "polaris branch create/promote/drop" > commands in the CLI. No persistence or REST changes, works on every > backend, > and users get atomic multi-table promotion in the next release. It also > tells > us which of B's limits people actually hit. > - Design C here on the list, starting with the two questions that decide > the > rest: > 1. How does a client select a branch so engines don't change? A warehouse > suffix like "sales@backfill", a prefix segment from /v1/config, or a > header? > 2. How do we keep GC safe with several heads? > And a broader one: is NoSQL the intended long-term default persistence? A > NoSQL-only feature is much easier to accept if it is. > > I wrote a longer comparison with the details and a table, and can share it > as a > doc if that's useful. If B sounds reasonable, I can start on the CLI part > and > the docs while we work out C. > > Thanks, > Ankur > > On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <[email protected]> > wrote: > > > Hi all, > > > > I'd like to start a discussion on catalog-level branching, tracked in > > https://www.google.com/url?q=https://github.com/apache/ > > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several users > > have asked for it > > there, and Dmitri suggested taking it to the list. Below is a first > > proposal. I'd > > much rather shape it with you than arrive with a finished design, so > > please poke > > holes in it. > > > > 1. The problem > > > > Iceberg already lets you branch a single table: a branch is just another > > snapshot > > ref in the table metadata. That works well for one table, but teams that > > run > > backfills, migrations or "write-audit-publish" pipelines across many > > tables hit > > three walls: > > > > - Creating or merging a branch across N tables takes N separate calls. > > - Promotion isn't atomic. While tables are merged one by one, readers can > > see > > some tables with the new data and others without it. With hundreds of > > tables, > > that window is long, and a failure halfway through leaves the catalog in > a > > mixed > > state someone has to repair by hand. > > - There's no cheap way to ask "which tables have branch X?" without > > loading every > > table's metadata. > > > > Today the answer is either to adopt Nessie, or to script it and accept > the > > non-atomic window. > > > > 2. What I'm proposing > > > > A catalog branch: a named, Polaris-managed group of per-table Iceberg > > branches > > that is created and promoted as a unit. > > > > polaris branch create backfill-2026 --from main --namespace sales > > ... jobs write to the "backfill-2026" branch of each table as usual ... > > polaris branch promote backfill-2026 --into main # all tables, or none > > > > Design goals: > > > > - Engines don't change. Writers and readers use the standard Iceberg > > branch API > > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref" > > field > > of REST commits). Nothing new is needed in Spark, Flink, Trino or > > PyIceberg. > > - Build on what Polaris already has. Per-table state stays in Iceberg > > metadata, > > and promotion reuses the transaction workspace that commitTransaction > > already > > uses to apply changes to several tables as one atomic metastore update. > > - Keep the first version small and safe. Fast-forward promotion only, > > explicit > > failure on conflict, no merge of diverged histories. > > > > 3. How it would work > > > > 3.1 A new entity > > > > A CATALOG_BRANCH entity, scoped to a catalog, holding: > > > > - the branch name, > > - its scope (a namespace, optionally recursive, or an explicit list of > > tables), > > - for each member table: the table id and the snapshot id of main when > the > > branch was created (the base), > > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED). > > > > Keeping membership in the entity answers "which tables have branch X?" > > with one > > read, instead of loading every table's metadata. > > > > 3.2 Create > > > > For every table in scope, add an Iceberg branch ref pointing at that > > table's > > current main snapshot, and write the CATALOG_BRANCH entity. All of it > goes > > through the transaction workspace, so either every table gets the ref and > > the > > entity exists, or nothing changes. > > > > 3.3 Write > > > > Nothing new. Jobs commit to the table's branch ref through the normal > REST > > updateTable path. Polaris only needs to reject commits that would move a > > catalog branch's ref outside of Polaris' control in ways that break > > promotion > > (for example deleting the ref). Whether that check is worth having in v1 > > is one > > of the questions below. > > > > 3.4 Promote > > > > For every member table, in one transaction: > > > > 1. load the current metadata, > > 2. check that main still points at the recorded base snapshot > (fast-forward > > check), > > 3. set main to the branch head, > > 4. optionally remove the branch ref. > > > > If any table fails the check, nothing is promoted, and the response lists > > the > > tables that moved since the branch was created. Readers either see the > old > > state > > of every table or the new state of every table. > > > > 3.5 Drop > > > > Remove the branch refs from member tables and mark the entity DROPPED. > > Snapshot > > expiry then cleans up the unreferenced data as it does today. > > > > 3.6 API and authorization > > > > A Polaris REST extension under the catalog API (create, list, get, > > promote, drop), > > plus CLI commands. Two options for authorization: new privileges > > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring > > TABLE_WRITE_DATA > > on every member table. The second is simpler, but slow to check for very > > large > > branches. > > > > 4. Edge cases I've thought about so far > > > > - Tables created in the namespace after the branch exists: not members in > > v1. > > They can be added explicitly. > > - A member table dropped or renamed while the branch is active: promotion > > fails > > with a clear error. Dropping the branch still works. > > - Schema changes made on the branch: carried by the branch snapshots, so > > promoting them is part of the same atomic switch. > > - Concurrent promotions of two branches that touch the same table: the > > second one > > fails the fast-forward check for that table, so nothing is promoted > twice. > > - Federated and static-facade catalogs: not supported, same as > > commitTransaction > > today. > > > > 5. Out of scope for a first version > > > > - Merging histories that have diverged (rebase or cherry-pick across > > tables). > > - Tags, commit logs and time travel at the catalog level, Nessie style. > > - Adding this to the Iceberg REST spec. If it proves useful, I'd be glad > to > > propose a generalized version to the Iceberg community later. > > > > 6. Questions for the community > > > > 1. Atomicity across backends: promotion depends on a multi-entity > > compare-and-set. Does every persistence backend (relational JDBC, NoSQL, > > DynamoDB) give the same guarantee here, or does a backend need to > advertise > > support? > > 2. How large should a single promotion be allowed to get? With thousands > of > > tables, the transaction gets big. Should there be a limit, or a two-phase > > approach? > > 3. Should Polaris protect a catalog branch's refs from direct deletion > > through > > the REST API? > > 4. Is a new entity type the right home, or would you prefer a > > property-based > > approach on namespaces? > > 5. Any appetite for aligning the vocabulary with Nessie's, given the > > earlier > > discussions about the two projects? > > > > I'm happy to present this at the next community sync and to do the > > implementation in small PRs: entity and persistence first, then create > and > > drop, > > then promote, then the CLI and docs. > > > > Thanks, > > Ankur Kotwal > > >
