Hi all, As promised, here's a design doc for catalog-level branches (option C), open for comments:
https://www.google.com/url?q=https://docs.google.com/document/d/1jLFkivxRZ0i4VHI-7G8olLQXk1sGAkUIFz_1xLGEJRA/edit&source=gmail&ust=1791530042497000&sa=E The short version, so the main points are in the archive too: - A branch is a second named pointer, cat/<id>/heads/<branch>, next to heads/main, plus an entry in a new per-catalog registry (cat/<id>/branches), since references can't be listed. Creating a branch, and publishing it while main hasn't moved, are each one compare-and-swap, however many tables the catalog has. - Clients pick a branch through the prefix (warehouse "lake@backfill", resolved by CatalogPrefixParser) or a Polaris-Branch header, so engines don't change. Catalog names can already contain "@", so a full-name match wins. - Version 1 publishes only when main hasn't moved; merging comes later. - Cleanup is the part that decides safety. Polaris objects: maintenance walks every registered pointer. Iceberg files: while a table is on more than one branch, Polaris rejects remove-snapshots commits (REMOVE_TABLE_SNAPSHOTS), sets gc.enabled=false once the table diverges (Iceberg's expiry and orphan-file actions refuse to run then), and skips purge on drop. Later, a catalog-side cleanup service enforcing the orphan-file-removal policy across all branches replaces that, as Dmitri suggested. - It needs the NoSQL model; with NoSQL on JDBC, RDBMS users get it too. On the classic JDBC persistence, the table-branch recipe from #5740 remains the way. - The plan is six small PRs behind a feature flag, starting with making the persistence branch-aware with no behavior change. The questions I'd most like input on are at the end of the doc, in particular whether blocking destructive maintenance on branched tables is acceptable for a first version. If it's useful, I'm happy to walk through it at the community sync on October 15. Thanks, Ankur On Wed, Oct 07, 2026 11:42 PM, Ankur Kotwal <[email protected]> wrote: > Hi Dmitri, > > Thanks, that's very helpful. > > Option B: understood, and thanks for the pointer to the multi-entity > thread. > The recipe only moves the refs of the tables it commits, and each of those > is > written with a check against the version that was read, so it doesn't > depend on > validations involving other entities. I've opened a test + doc PR as you > suggested: https://www.google.com/url?q=https://github.com/apache/ > polaris/pull/5740&source=gmail&ust=1791528160935000&sa=E. Happy to add > the same > test against the NoSQL persistence if that helps. > > Option C: I'm glad to explore it from the start, and NoSQL on JDBC would > make it > much more broadly useful. Since GC is the part that decides whether > branches are > safe, I'd like to start there. My current thinking: > > - Polaris objects: branches need a registry the maintenance service can > walk, > since references can't be listed, and every head is retained with the same > CEL rules as main. > - Iceberg files: until there is a catalog-side maintenance service, a > table that > exists on more than one head gets gc.enabled=false. Iceberg's > RemoveSnapshots > and the Spark orphan-file action both refuse to run then ("GC is disabled > (deleting files may corrupt other tables)"), and Polaris could reject > snapshot-removing commits on those tables as a second guard. Later, a > catalog-side service enforcing the existing orphan-file-removal policy > across > all heads would replace that, which matches your point about GC becoming a > catalog concern. > > I'll write this up as a short design doc, covering branch selection (prefix > and header) too, and share it here. If it's useful, I can walk through it > at the > next community sync. > > Thanks, > Ankur > > On Wed, Oct 07, 2026 06:30 PM, Dmitri Bourlatchkov <[email protected]> > wrote: > >> Hi Ankur, >> >> Thanks again for taking this feature discussion to the dev ML! >> >> Re: Option B: atomic multi-entity changes have some issues with the >> current >> Persistence framework. Basic operations work well, but there are concerns >> with finer points. There is a rather long discussion about that in [1]. >> >> I need some time to consider this idea. Let's see what other people think. >> >> If you could contribute a test + doc PR to Polaris for this idea, it would >> be great! >> >> Re: Option C: Catalog-level branches are indeed natural in the NoSQL >> Persistence. I'd personally prefer to explore this option from the start >> (but I do not insist). >> >> If there's interest in supporting the NoSQL model over JDBC, I believe it >> is quite possible to implement (despite the NoSQL name). That will open >> native catalog branches to RDBMS-based environments too. >> >> The only operational drawback would be having to run NoSQL repository >> maintenance [2] even on RDBMS, but I think it is quite an acceptable >> inconvenience for the power of the feature as a whole. >> >> The traditional JDBC persistence and the proposed "Atomic DynamoDB" [5615] >> persistence will remain as the "classical" options, of course. Users >> interested in catalog-level branches will have the option to use NoSQL >> with >> MongoDB (current codebase), DynamoDB [5725] or JDBC (future PR). >> >> I certainly consider the NoSQL Persistence framework to be a long-term >> feature. Specific backend support depends on user interest, I think, but >> the range of possibilities virtually covers the majority of existing >> databases. I'd also add that onboarding a new backend is rather simple. >> All >> the hard tasks are handled in common code. >> >> Re: Question 1: both the "prefix" and the "header" approaches can work, I >> think. Even at the same time. >> >> Re: Question 2 (GC): IMHO, GC becomes a catalog-side concern, and that is >> a >> good thing. Polaris already has some relevant concepts, >> e.g. OrphanFileRemovalPolicyContent [1261]. >> >> IIRC, there were some prior discussions about an external "table >> maintenance service" that could enforce those policies (I could not find a >> specific dev thread, though). >> >> [1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp >> >> [2] >> https://polaris.apache.org/releases/1.8.0/metastores/nosql-m >> ongodb/#regular-maintenance >> >> [1261] https://github.com/apache/polaris/pull/1261 >> >> [5615] https://github.com/apache/polaris/issues/5615 >> >> [5725] https://github.com/apache/polaris/pull/5725 >> >> Cheers, >> Dmitri. >> >> On Wed, Oct 7, 2026 at 2:43 PM Ankur Kotwal <[email protected]> >> wrote: >> >> > Hi all, >> > >> > Following up on my own proposal, since Dmitri questioned the new entity >> > type on >> > the issue and he had a point. I spent some time in the code, and there >> are >> > two >> > things already in Polaris that I underestimated. Here's how I'd see the >> > options >> > now. >> > >> > What exists today >> > >> > 1. commitTransaction is atomic on both persistence families. The handler >> > stages >> > every table change and then writes all entities in one go: relational >> JDBC >> > in >> > one database transaction with version checks, NoSQL as one commit >> against >> > the >> > catalog's named pointer. >> > >> > 2. The NoSQL persistence is already shaped like Git: a catalog's >> content is >> > an >> > immutable state object behind a named pointer, changes are commits with >> a >> > single CAS, and there's a commit log. The pointer is literally called >> > cat/<id>/heads/main. >> > >> > Three ways to get to "promote many tables atomically" >> > >> > A. My original proposal: a CATALOG_BRANCH entity on top of per-table >> > Iceberg >> > branch refs. Looking at it again, the entity mostly duplicates what the >> > refs >> > already say, and promotion still touches every member table. I'd drop >> it. >> > >> > B. No new server concept at all. A client creates a branch ref on each >> > table in >> > one transactions/commit, writes to it as usual, and promotes with one >> more >> > transactions/commit that, per table, asserts main is still at its base >> > snapshot (assert-ref-snapshot-id) and moves main to the branch head >> > (set-snapshot-ref). Either every table moves or none does, on every >> > backend, >> > today. What it can't do comes from Iceberg table branches: schemas are >> per >> > table, so a schema change on the branch is visible on main; tables, >> views >> > and >> > namespaces can't be created or dropped "on a branch"; and the client >> has to >> > know the member list. >> > >> > I checked that this works with a small test against the REST handler >> > (the in-memory metastore the service tests use): three tables, branch >> > created in one commit, written to, promoted in one commit. When main of >> one >> > table was moved by another writer after branching, the promotion was >> > rejected and none of the three tables changed. Happy to turn it into a >> > proper test and a docs page. >> > >> > C. Real catalog branches as more named pointers, >> cat/<id>/heads/<branch>, >> > on the >> > NoSQL persistence. Creating a branch is one new pointer; promoting is a >> > single >> > CAS when main hasn't moved, for any number of tables; schema changes and >> > created or dropped tables live on the branch; and it's the same model as >> > Nessie. It only works on NoSQL, and it needs real work: the reference >> name >> > threaded through the places that assume main (NoSqlMetaStore, References >> > and >> > the Table/Namespace/Policy mappings), a way for clients to pick a >> branch, >> > and >> > above all garbage collection. Today CatalogRetainedIdentifier only >> retains >> > what heads/main reaches, and Iceberg snapshot expiry or orphan-file >> removal >> > run on main can delete files that only a branch's table metadata >> > references. >> > Nessie needed a dedicated GC for this, so I'd want that settled before >> any >> > code. >> > >> > What I'd suggest >> > >> > - Do B now: a documented recipe plus "polaris branch >> create/promote/drop" >> > commands in the CLI. No persistence or REST changes, works on every >> > backend, >> > and users get atomic multi-table promotion in the next release. It also >> > tells >> > us which of B's limits people actually hit. >> > - Design C here on the list, starting with the two questions that decide >> > the >> > rest: >> > 1. How does a client select a branch so engines don't change? A >> warehouse >> > suffix like "sales@backfill", a prefix segment from /v1/config, or a >> > header? >> > 2. How do we keep GC safe with several heads? >> > And a broader one: is NoSQL the intended long-term default persistence? >> A >> > NoSQL-only feature is much easier to accept if it is. >> > >> > I wrote a longer comparison with the details and a table, and can share >> it >> > as a >> > doc if that's useful. If B sounds reasonable, I can start on the CLI >> part >> > and >> > the docs while we work out C. >> > >> > Thanks, >> > Ankur >> > >> > On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <[email protected]> >> > wrote: >> > >> > > Hi all, >> > > >> > > I'd like to start a discussion on catalog-level branching, tracked in >> > > https://www.google.com/url?q=https://github.com/apache/ >> > > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several >> users >> > > have asked for it >> > > there, and Dmitri suggested taking it to the list. Below is a first >> > > proposal. I'd >> > > much rather shape it with you than arrive with a finished design, so >> > > please poke >> > > holes in it. >> > > >> > > 1. The problem >> > > >> > > Iceberg already lets you branch a single table: a branch is just >> another >> > > snapshot >> > > ref in the table metadata. That works well for one table, but teams >> that >> > > run >> > > backfills, migrations or "write-audit-publish" pipelines across many >> > > tables hit >> > > three walls: >> > > >> > > - Creating or merging a branch across N tables takes N separate calls. >> > > - Promotion isn't atomic. While tables are merged one by one, readers >> can >> > > see >> > > some tables with the new data and others without it. With hundreds of >> > > tables, >> > > that window is long, and a failure halfway through leaves the catalog >> in >> > a >> > > mixed >> > > state someone has to repair by hand. >> > > - There's no cheap way to ask "which tables have branch X?" without >> > > loading every >> > > table's metadata. >> > > >> > > Today the answer is either to adopt Nessie, or to script it and accept >> > the >> > > non-atomic window. >> > > >> > > 2. What I'm proposing >> > > >> > > A catalog branch: a named, Polaris-managed group of per-table Iceberg >> > > branches >> > > that is created and promoted as a unit. >> > > >> > > polaris branch create backfill-2026 --from main --namespace sales >> > > ... jobs write to the "backfill-2026" branch of each table as usual >> ... >> > > polaris branch promote backfill-2026 --into main # all tables, or none >> > > >> > > Design goals: >> > > >> > > - Engines don't change. Writers and readers use the standard Iceberg >> > > branch API >> > > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref" >> > > field >> > > of REST commits). Nothing new is needed in Spark, Flink, Trino or >> > > PyIceberg. >> > > - Build on what Polaris already has. Per-table state stays in Iceberg >> > > metadata, >> > > and promotion reuses the transaction workspace that commitTransaction >> > > already >> > > uses to apply changes to several tables as one atomic metastore >> update. >> > > - Keep the first version small and safe. Fast-forward promotion only, >> > > explicit >> > > failure on conflict, no merge of diverged histories. >> > > >> > > 3. How it would work >> > > >> > > 3.1 A new entity >> > > >> > > A CATALOG_BRANCH entity, scoped to a catalog, holding: >> > > >> > > - the branch name, >> > > - its scope (a namespace, optionally recursive, or an explicit list of >> > > tables), >> > > - for each member table: the table id and the snapshot id of main when >> > the >> > > branch was created (the base), >> > > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED). >> > > >> > > Keeping membership in the entity answers "which tables have branch X?" >> > > with one >> > > read, instead of loading every table's metadata. >> > > >> > > 3.2 Create >> > > >> > > For every table in scope, add an Iceberg branch ref pointing at that >> > > table's >> > > current main snapshot, and write the CATALOG_BRANCH entity. All of it >> > goes >> > > through the transaction workspace, so either every table gets the ref >> and >> > > the >> > > entity exists, or nothing changes. >> > > >> > > 3.3 Write >> > > >> > > Nothing new. Jobs commit to the table's branch ref through the normal >> > REST >> > > updateTable path. Polaris only needs to reject commits that would >> move a >> > > catalog branch's ref outside of Polaris' control in ways that break >> > > promotion >> > > (for example deleting the ref). Whether that check is worth having in >> v1 >> > > is one >> > > of the questions below. >> > > >> > > 3.4 Promote >> > > >> > > For every member table, in one transaction: >> > > >> > > 1. load the current metadata, >> > > 2. check that main still points at the recorded base snapshot >> > (fast-forward >> > > check), >> > > 3. set main to the branch head, >> > > 4. optionally remove the branch ref. >> > > >> > > If any table fails the check, nothing is promoted, and the response >> lists >> > > the >> > > tables that moved since the branch was created. Readers either see the >> > old >> > > state >> > > of every table or the new state of every table. >> > > >> > > 3.5 Drop >> > > >> > > Remove the branch refs from member tables and mark the entity DROPPED. >> > > Snapshot >> > > expiry then cleans up the unreferenced data as it does today. >> > > >> > > 3.6 API and authorization >> > > >> > > A Polaris REST extension under the catalog API (create, list, get, >> > > promote, drop), >> > > plus CLI commands. Two options for authorization: new privileges >> > > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring >> > > TABLE_WRITE_DATA >> > > on every member table. The second is simpler, but slow to check for >> very >> > > large >> > > branches. >> > > >> > > 4. Edge cases I've thought about so far >> > > >> > > - Tables created in the namespace after the branch exists: not >> members in >> > > v1. >> > > They can be added explicitly. >> > > - A member table dropped or renamed while the branch is active: >> promotion >> > > fails >> > > with a clear error. Dropping the branch still works. >> > > - Schema changes made on the branch: carried by the branch snapshots, >> so >> > > promoting them is part of the same atomic switch. >> > > - Concurrent promotions of two branches that touch the same table: the >> > > second one >> > > fails the fast-forward check for that table, so nothing is promoted >> > twice. >> > > - Federated and static-facade catalogs: not supported, same as >> > > commitTransaction >> > > today. >> > > >> > > 5. Out of scope for a first version >> > > >> > > - Merging histories that have diverged (rebase or cherry-pick across >> > > tables). >> > > - Tags, commit logs and time travel at the catalog level, Nessie >> style. >> > > - Adding this to the Iceberg REST spec. If it proves useful, I'd be >> glad >> > to >> > > propose a generalized version to the Iceberg community later. >> > > >> > > 6. Questions for the community >> > > >> > > 1. Atomicity across backends: promotion depends on a multi-entity >> > > compare-and-set. Does every persistence backend (relational JDBC, >> NoSQL, >> > > DynamoDB) give the same guarantee here, or does a backend need to >> > advertise >> > > support? >> > > 2. How large should a single promotion be allowed to get? With >> thousands >> > of >> > > tables, the transaction gets big. Should there be a limit, or a >> two-phase >> > > approach? >> > > 3. Should Polaris protect a catalog branch's refs from direct deletion >> > > through >> > > the REST API? >> > > 4. Is a new entity type the right home, or would you prefer a >> > > property-based >> > > approach on namespaces? >> > > 5. Any appetite for aligning the vocabulary with Nessie's, given the >> > > earlier >> > > discussions about the two projects? >> > > >> > > I'm happy to present this at the next community sync and to do the >> > > implementation in small PRs: entity and persistence first, then create >> > and >> > > drop, >> > > then promote, then the CLI and docs. >> > > >> > > Thanks, >> > > Ankur Kotwal >> > > >> > >> >
