cshuo commented on code in PR #20459: URL: https://github.com/apache/hudi/pull/20459#discussion_r4449861952
########## rfc/rfc-116/rfc-116.md: ########## @@ -0,0 +1,438 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-116: Support Branches and Tags + +## Proposers + +- @cshuo +- @danny0405 + +## Approvers + +- TBD + +## Status + +RFC reservation: [#20456](https://github.com/apache/hudi/pull/20456) + +## Abstract + +This RFC adds writable branches and read-only tags to Hudi. The data table and a single metadata table (MDT) share immutable files across branches, while each branch has its own timelines. MDT reads follow the selected branch’s timeline. A working branch can publish to main through strict fast-forward. Writing the Data-side `_SUCCESS` publishes the paired Data/MDT state to main. Tags protect paired historical Data/MDT states for time-travel reads and writable branch creation. + +## Background + +Branches let users write and validate changes before exposing them on main. Tags name historical states and retain the files needed to read them; ordinary time travel alone does not provide that protection. + +This RFC covers an initial subset of [apache/hudi#19066](https://github.com/apache/hudi/issues/19066): isolated batch writes, repeatable historical reads and publication to main. Strict fast-forward requires inherited logical write history; completed clustering, compaction and log compaction only change physical layout. If main and audit independently change data after a common ancestor, pausing main cannot make audit publishable; merging divergent changes is out of scope. + +| Area | Scope | +|---|---| +| Engine | Spark batch | +| Data | COW and MOR, table versions 8/9/10 | +| MDT | Enabled MOR table; one physical root with shared index files | +| Definitions | Fixed schema, keys, partition layout, merge configuration and index definitions | +| Branch creation | main's latest completed commit or an unexpired tag on main | +| Publication | Working branch → main, strict fast-forward | + +Out of scope: branch creation from working branches, versions below 8, Flink, streaming/incremental/CDC reads, bootstrap, merge/rebase, independent format upgrades per branch, external MDT service managers and non-main catalog sync. + +## Implementation + +### 1. Shared files and branch views + +Data and MDT branches with the same name form a pair. Each has its own configuration and active/archived timeline. Data files and MDT base/log files are shared; new writes produce immutable files. Branches can update the same inherited fileId independently. + +A new branch takes history and files from main's latest completed commit or a tag on main, and configuration from main. Fixed table definitions must agree across branches and cannot change after creation. Branch-local runtime tuning may differ. + +- **Data FSView:** Use the selected branch's MDT FILES, timeline and replace history. Directory listing alone does not establish membership. +- **MDT FSView:** MDT has no nested MDT. List physical files, then filter their requested instants against the selected MDT branch's active and archived timeline **before** constructing file groups/slices. An instant older than the active timeline start is not proof of membership. + +The design enforces three invariants: + +1. Reads and writes stay within the selected branch's paired Data/MDT state. +2. Publication preserves main's logical write history and tag/savepoint protections. +3. Shared files remain until all protections from branches, savepoints and pending instants allow deletion. + +### 2. Branches and metadata layout + +Main is the default branch. Initially, its metadata stays in the Data and MDT root `.hoodie/` directories. Working branches use `branches/<branch-name>/`; publications use `branches/publications/<publication-id>/` on both sides. Publication directories are created only when a working branch publishes to main. Branch discovery excludes the reserved `publications/` directory. +In the layouts below, `timeline/` includes active and archived history. + +```text +<table>/.hoodie/ + hoodie.properties + timeline/ # initial main Data timeline + tags/<tag-name>.json # initial main tags + branches/ + publications/<publication-id>/ # published main Data metadata + _SUCCESS # written last to publish the paired Data/MDT state + hoodie.properties + timeline/ + tags/ + <branch-name>/ # working-branch Data metadata + branch.json + hoodie.properties + timeline/ + tags/ + metadata/ + files/ + record_index/ + secondary_index_<index-name>/ + .hoodie/ + hoodie.properties + timeline/ # initial main MDT timeline + branches/ + publications/<publication-id>/ # paired published main MDT metadata + hoodie.properties + timeline/ + <branch-name>/ # paired working-branch MDT metadata + branch.json + hoodie.properties + timeline/ +``` + +#### Main resolution + +A `publicationId` has the form `p<N>`. Under the table lock, scan all Data and MDT publication directories, including unfinished preparations, and choose the largest numeric `N` plus one (`p1` if none exist). Reserve the number before releasing the lock by creating the Data-side `hoodie.properties` without overwrite; this also makes the prefix visible on object stores. Keep the current successful publication. A failed preparation's number may be reused only after its operation and storage requests have stopped and both directories have been fully removed under table coordination. Review Comment: Added definitions for publication and `_SUCCESS` in Terminology, followed by a high-level explanation of how the marker activates the prepared Data and MDT histories together before the detailed publication flow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
