danny0405 commented on code in PR #20459: URL: https://github.com/apache/hudi/pull/20459#discussion_r4420201333
########## rfc/rfc-116/rfc-116.md: ########## @@ -0,0 +1,438 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-116: Support Branches and Tags + +## Proposers + +- @cshuo +- @danny0405 + +## Approvers + +- TBD + +## Status + +RFC reservation: [#20456](https://github.com/apache/hudi/pull/20456) + +## Abstract + +This RFC adds writable branches and read-only tags to Hudi. The data table and a single metadata table (MDT) share immutable files across branches, while each branch has its own timelines. MDT reads follow the selected branch’s timeline. A working branch can publish to main through strict fast-forward. Writing the Data-side `_SUCCESS` publishes the paired Data/MDT state to main. Tags protect paired historical Data/MDT states for time-travel reads and writable branch creation. + +## Background + +Branches let users write and validate changes before exposing them on main. Tags name historical states and retain the files needed to read them; ordinary time travel alone does not provide that protection. + +This RFC covers an initial subset of [apache/hudi#19066](https://github.com/apache/hudi/issues/19066): isolated batch writes, repeatable historical reads and publication to main. Strict fast-forward requires inherited logical write history; completed clustering, compaction and log compaction only change physical layout. If main and audit independently change data after a common ancestor, pausing main cannot make audit publishable; merging divergent changes is out of scope. + +| Area | Scope | +|---|---| +| Engine | Spark batch | +| Data | COW and MOR, table versions 8/9/10 | +| MDT | Enabled MOR table; one physical root with shared index files | +| Definitions | Fixed schema, keys, partition layout, merge configuration and index definitions | +| Branch creation | main's latest completed commit or an unexpired tag on main | +| Publication | Working branch → main, strict fast-forward | + +Out of scope: branch creation from working branches, versions below 8, Flink, streaming/incremental/CDC reads, bootstrap, merge/rebase, independent format upgrades per branch, external MDT service managers and non-main catalog sync. + +## Implementation + +### 1. Shared files and branch views + +Data and MDT branches with the same name form a pair. Each has its own configuration and active/archived timeline. Data files and MDT base/log files are shared; new writes produce immutable files. Branches can update the same inherited fileId independently. + +A new branch takes history and files from main's latest completed commit or a tag on main, and configuration from main. Fixed table definitions must agree across branches and cannot change after creation. Branch-local runtime tuning may differ. + +- **Data FSView:** Use the selected branch's MDT FILES, timeline and replace history. Directory listing alone does not establish membership. +- **MDT FSView:** MDT has no nested MDT. List physical files, then filter their requested instants against the selected MDT branch's active and archived timeline **before** constructing file groups/slices. An instant older than the active timeline start is not proof of membership. + +The design enforces three invariants: + +1. Reads and writes stay within the selected branch's paired Data/MDT state. +2. Publication preserves main's logical write history and tag/savepoint protections. +3. Shared files remain until all protections from branches, savepoints and pending instants allow deletion. + +### 2. Branches and metadata layout + +Main is the default branch. Initially, its metadata stays in the Data and MDT root `.hoodie/` directories. Working branches use `branches/<branch-name>/`; publications use `branches/publications/<publication-id>/` on both sides. Publication directories are created only when a working branch publishes to main. Branch discovery excludes the reserved `publications/` directory. +In the layouts below, `timeline/` includes active and archived history. + +```text +<table>/.hoodie/ + hoodie.properties + timeline/ # initial main Data timeline + tags/<tag-name>.json # initial main tags + branches/ + publications/<publication-id>/ # published main Data metadata + _SUCCESS # written last to publish the paired Data/MDT state + hoodie.properties + timeline/ + tags/ + <branch-name>/ # working-branch Data metadata + branch.json + hoodie.properties + timeline/ + tags/ + metadata/ + files/ + record_index/ + secondary_index_<index-name>/ + .hoodie/ + hoodie.properties + timeline/ # initial main MDT timeline + branches/ + publications/<publication-id>/ # paired published main MDT metadata + hoodie.properties + timeline/ + <branch-name>/ # paired working-branch MDT metadata + branch.json + hoodie.properties + timeline/ +``` + +#### Main resolution + +A `publicationId` has the form `p<N>`. Under the table lock, scan all Data and MDT publication directories, including unfinished preparations, and choose the largest numeric `N` plus one (`p1` if none exist). Reserve the number before releasing the lock by creating the Data-side `hoodie.properties` without overwrite; this also makes the prefix visible on object stores. Keep the current successful publication. A failed preparation's number may be reused only after its operation and storage requests have stopped and both directories have been fully removed under table coordination. + +Before the first successful publication, main uses the paired root `.hoodie/` metadata. Preparation directories without `_SUCCESS` do not change that selection. Once published, main uses the greatest **numeric** publication number with a Data-side `_SUCCESS`; that ID selects both metadata directories. Scan all listing pages and check Data-side `_SUCCESS` files under the read-capture lock. Publication failure leaves the current selection unchanged; missing or unreadable selected metadata is an error, not permission to revert to an older state. + +Ordinary commits, maintenance and tag changes update the selected root or publication metadata. They do not create publication directories or change `_SUCCESS` or the publication ID. A publication ID identifies metadata directories, not a fixed data snapshot. + +#### Create and drop + +Hold the table lock throughout branch creation: + +1. Select main's current completed Data/MDT state, or the historical state of `from_tag` on main. Record the inherited main Data instant as `baseInstant` (null if main has no completed commits). Reject missing/expired tags and requested/inflight instants that could change this state. +2. Copy the branch metadata listed below into the new paired directories, using main's configuration. Share Data files and MDT index files, including RLI/SI. Missing metadata or copy failures abort creation. Do not inherit pending table services as runnable work. +3. Confirm metadata copying has completed on both sides, then write the MDT `branch.json` first and the Data `branch.json` last. Release the lock after activation or failure. + +Both `branch.json` files carry the same never-reused `branchId`. The Data `branch.json` makes the branch discoverable only when the MDT identity matches. Inherited tags remain protected independently of their source. Replan services under new instants. + +For `from_tag`: + +- Select the historical Data/MDT state protected by the paired savepoints. Copy the existing archived timelines, including their manifests, and exclude writes, replacements and table services after the tag instant from the active timelines. +- Copy matching schema history and index definitions, the selected tag and both savepoints. Exclude unrelated tags/savepoints; reject definitions incompatible with main's fixed configuration. +- Share the protected files without rebuilding data or indexes; the activated branch no longer depends on the source tag's protection. + +The following metadata is maintained per branch. Copy applicable contents for each Data/MDT metadata root when creating a branch or preparing a publication; absent optional directories need not be created. + +| Metadata | Contents and handling | +|---|---| +| `hoodie.properties` | Complete configuration: use main's configuration for branch creation and the captured source configuration for publication. Adjust metadata paths for the target. | +| `timeline/` | Completed history, including savepoints and archived history with its manifests and referenced files. Preserve a consistent captured state; do not inherit pending instants as runnable work. | +| `.schema/` | Completed schema history matching the captured commits, including historical schemas and field IDs. Exclude pending files and later source state. | +| `.index_defs/index.json` or the configured index-definition file | Actual index definitions and versions, with configuration pointing to the target copy. | +| `tags/` | Valid tag definitions. Paired Data/MDT savepoints are included in their respective `timeline/` directories. Applies to Data metadata. | Review Comment: why tags are maintained as per branch, should they be on the same hierarchy with branches? the only difference is mutability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
