SteNicholas opened a new issue, #351: URL: https://github.com/apache/paimon-cpp/issues/351
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation In Java Paimon a commit is aimed at a branch: `SnapshotCommit#commit` takes `branch`, `CatalogSnapshotCommit` re-targets the table identifier at it (`new Identifier(database, table, branch)`), and `RenamingSnapshotCommit` writes the snapshot file into that branch's snapshot directory (`snapshotManager.copyWithBranch(branch)`). #326 ported the rest of `CatalogSnapshotCommit` but left the branch out, because the C++ request carried no identifier for a branch to apply to. Paimon C++ cannot commit to a branch at all today: - `SnapshotCommit::Commit()` has no branch parameter, so `CatalogSnapshotCommit` always commits to the identifier its context was built with, and `RenamingSnapshotCommit` always writes to the snapshot directory of the manager it was built with. - `FileStoreCommit::Create()` builds the `SchemaManager` and the `SnapshotManager` for the main branch whatever the `branch` option says, while `FileStoreWrite::Create()` is already branch-aware. A writer aimed at `dev` therefore reads that branch's schema and snapshots while the commit publishes on main. - `CommitContextBuilder` and `WriteContextBuilder` refuse a non-main branch outright once a catalog is configured ("a commit through a catalog reads the main branch of the table"), so a catalog table's branch can be neither written nor committed. - `ReadContext::GetBranch()` reflects only `WithBranch()`, so a read aimed at a branch through the `branch` option alone - which is how a scan of one is aimed, and how `RestCatalog` reports a branch table's options - takes the main branch's schema, while the splits, the path factory and the split readers follow the option. The integration tests show the gap: a branch is fabricated by copying the `schema` and `snapshot` directories, and `test/inte/pk_compaction_inte_test.cpp` writes and compacts on a branch but can only verify the commit messages, "since we don't support branch commit and scan". ### Solution Port the branch through the snapshot-commit layer, then resolve the branch once for every entry point. - `SnapshotCommit::Commit(base_snapshot_uuid, snapshot, branch, statistics)`, as in Java. `CatalogSnapshotCommit` commits to `Identifier(database, data table name, branch)` so the catalog answers for the branch, with the main branch staying the bare table name, matched ignoring case as the identifier the Java client builds is. `RenamingSnapshotCommit` writes the snapshot file and the `LATEST` hint through a manager of the branch asked for, mirroring `copyWithBranch`. `FileStoreCommitImpl` passes `snapshot_manager_->Branch()`: the branch whose metadata the commit reads is the one it publishes on, which is the convention its real-time offsets already follow. - New `Identifier(database, table, branch)`, equivalent to the Java constructor, building the `<table>$branch_<branch>` object name. - New `BranchManager::ResolveBranch(identifier, options, asked_branch, operation)`: the identifier, the `branch` option and `WithBranch()` may each name the branch, an absent or empty name is the main branch, and two different names are refused rather than silently chosen between. `CommitContextBuilder`, `WriteContextBuilder` and `ReadContextBuilder` all resolve through it, and with a catalog a non-main branch has to be named in the identifier, since the catalog is asked for the branch's schema and snapshot by identifier. - `FileStoreCommit::Create()` builds the `SchemaManager`, the format-table dispatch and the `SnapshotManager` for the resolved branch, and puts it back into the merged `CoreOptions` so that what reads through them reads the same branch - manifest entry cache keys are scoped by it. `FileStoreWrite::Create()` does the same, which also fixes a `WithBranch()` write whose compaction reads went through main-scoped options. ### Anything else? Behaviour changes: a file-system commit with `branch=dev` publishes to that branch instead of silently to main; `WithCatalog()` accepts `tbl$branch_dev`; `WriteContext::GetBranch()` and `ReadContext::GetBranch()` return the resolved branch, so `WithBranch("")` reads back as `main`. A branch keeps its own schema, snapshots and real-time offsets under `branch/branch-<name>` while data and manifests stay shared under the table path, as in Java. Two consequences worth documenting: the branch's schema is read from the table directory rather than from the catalog, so a branch committed to through a catalog needs its schema published there; and expiration walks the retained snapshots of the branch it commits to while the data files are shared with the other branches of the table. Public API touched: the new `Identifier` constructor, and the documented behaviour of `CommitContextBuilder::WithCatalog()`, `WriteContextBuilder::WithBranch()` and `ReadContextBuilder::WithBranch()`. No storage format or protocol change. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
