Hi all, I'd like to start a discussion on catalog-level branching, tracked in https://www.google.com/url?q=https://github.com/apache/polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several users have asked for it there, and Dmitri suggested taking it to the list. Below is a first proposal. I'd much rather shape it with you than arrive with a finished design, so please poke holes in it.
1. The problem Iceberg already lets you branch a single table: a branch is just another snapshot ref in the table metadata. That works well for one table, but teams that run backfills, migrations or "write-audit-publish" pipelines across many tables hit three walls: - Creating or merging a branch across N tables takes N separate calls. - Promotion isn't atomic. While tables are merged one by one, readers can see some tables with the new data and others without it. With hundreds of tables, that window is long, and a failure halfway through leaves the catalog in a mixed state someone has to repair by hand. - There's no cheap way to ask "which tables have branch X?" without loading every table's metadata. Today the answer is either to adopt Nessie, or to script it and accept the non-atomic window. 2. What I'm proposing A catalog branch: a named, Polaris-managed group of per-table Iceberg branches that is created and promoted as a unit. polaris branch create backfill-2026 --from main --namespace sales ... jobs write to the "backfill-2026" branch of each table as usual ... polaris branch promote backfill-2026 --into main # all tables, or none Design goals: - Engines don't change. Writers and readers use the standard Iceberg branch API (for example sales.orders.branch_backfill-2026 in Spark, or the "ref" field of REST commits). Nothing new is needed in Spark, Flink, Trino or PyIceberg. - Build on what Polaris already has. Per-table state stays in Iceberg metadata, and promotion reuses the transaction workspace that commitTransaction already uses to apply changes to several tables as one atomic metastore update. - Keep the first version small and safe. Fast-forward promotion only, explicit failure on conflict, no merge of diverged histories. 3. How it would work 3.1 A new entity A CATALOG_BRANCH entity, scoped to a catalog, holding: - the branch name, - its scope (a namespace, optionally recursive, or an explicit list of tables), - for each member table: the table id and the snapshot id of main when the branch was created (the base), - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED). Keeping membership in the entity answers "which tables have branch X?" with one read, instead of loading every table's metadata. 3.2 Create For every table in scope, add an Iceberg branch ref pointing at that table's current main snapshot, and write the CATALOG_BRANCH entity. All of it goes through the transaction workspace, so either every table gets the ref and the entity exists, or nothing changes. 3.3 Write Nothing new. Jobs commit to the table's branch ref through the normal REST updateTable path. Polaris only needs to reject commits that would move a catalog branch's ref outside of Polaris' control in ways that break promotion (for example deleting the ref). Whether that check is worth having in v1 is one of the questions below. 3.4 Promote For every member table, in one transaction: 1. load the current metadata, 2. check that main still points at the recorded base snapshot (fast-forward check), 3. set main to the branch head, 4. optionally remove the branch ref. If any table fails the check, nothing is promoted, and the response lists the tables that moved since the branch was created. Readers either see the old state of every table or the new state of every table. 3.5 Drop Remove the branch refs from member tables and mark the entity DROPPED. Snapshot expiry then cleans up the unreferenced data as it does today. 3.6 API and authorization A Polaris REST extension under the catalog API (create, list, get, promote, drop), plus CLI commands. Two options for authorization: new privileges (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring TABLE_WRITE_DATA on every member table. The second is simpler, but slow to check for very large branches. 4. Edge cases I've thought about so far - Tables created in the namespace after the branch exists: not members in v1. They can be added explicitly. - A member table dropped or renamed while the branch is active: promotion fails with a clear error. Dropping the branch still works. - Schema changes made on the branch: carried by the branch snapshots, so promoting them is part of the same atomic switch. - Concurrent promotions of two branches that touch the same table: the second one fails the fast-forward check for that table, so nothing is promoted twice. - Federated and static-facade catalogs: not supported, same as commitTransaction today. 5. Out of scope for a first version - Merging histories that have diverged (rebase or cherry-pick across tables). - Tags, commit logs and time travel at the catalog level, Nessie style. - Adding this to the Iceberg REST spec. If it proves useful, I'd be glad to propose a generalized version to the Iceberg community later. 6. Questions for the community 1. Atomicity across backends: promotion depends on a multi-entity compare-and-set. Does every persistence backend (relational JDBC, NoSQL, DynamoDB) give the same guarantee here, or does a backend need to advertise support? 2. How large should a single promotion be allowed to get? With thousands of tables, the transaction gets big. Should there be a limit, or a two-phase approach? 3. Should Polaris protect a catalog branch's refs from direct deletion through the REST API? 4. Is a new entity type the right home, or would you prefer a property-based approach on namespaces? 5. Any appetite for aligning the vocabulary with Nessie's, given the earlier discussions about the two projects? I'm happy to present this at the next community sync and to do the implementation in small PRs: entity and persistence first, then create and drop, then promote, then the CLI and docs. Thanks, Ankur Kotwal
