voonhous commented on code in PR #13152: URL: https://github.com/apache/hudi/pull/13152#discussion_r3568032440
########## rfc/rfc-94/rfc-94.md: ########## @@ -0,0 +1,662 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-94: Hudi Timeline User Interface (UI) + +## Proposers + +- @voonhous + +## Approvers + +- @danny0405 +- @rahil-c +- @yihua + +## Status + +JIRA: [HUDI-9315](https://issues.apache.org/jira/browse/HUDI-9315) + +## Abstract + +Hudi Timeline metadata is stored as timestamped files representing state transitions of actions like `commit`, +`deltacommit` and `compaction`. These files are accessible via the CLI or a file explorer, but it's hard to visualize +concurrent actions, spot missing transitions, or tell how long each step took. Debugging timeline issues by reading +filenames is tedious. + +This RFC proposes a UI-based timeline visualization tool that parses these metadata files, groups related actions, and +renders them in a time-ordered, interactive view. Users can track the lifecycle of each operation, see concurrency +patterns, and spot anomalies or long-running tasks. The implementation extends `hudi-timeline-service` with new `/v2/` +REST APIs and a static HTML + JavaScript frontend powered by [vis-timeline](https://github.com/visjs/vis-timeline), +served via Javalin's built-in static file serving with zero new Java compile-time dependencies. + +## Background + +Today, we rely on the CLI or direct filesystem inspection to understand timeline state through metadata files. These +files represent different actions (e.g., `deltacommit`, `compaction`) and their lifecycle states (`requested`, +`inflight`, `completed`), encoded in file names like: + +```shell +20250409102118815.deltacommit.inflight +20250409102118815.deltacommit.requested +20250409102118815_20250409102124339.deltacommit +20250409102121593.compaction.inflight +20250409102121593.compaction.requested +20250409102121593_20250409102122232.commit +20250409102124581.deltacommit.inflight +20250409102124581.deltacommit.requested +20250409102124581_20250409102125667.deltacommit +20250409102124612.compaction.inflight +20250409102124612.compaction.requested +20250409102124612_20250409102124892.commit +20250409102127348.deltacommit.inflight +20250409102127348.deltacommit.requested +20250409102127348_20250409102128481.deltacommit +20250409102127500.compaction.inflight +20250409102127500.compaction.requested +20250409102127500_20250409102127721.commit +``` + +This works, but has a few problems: + +1. No visibility into concurrency + - Multiple actions (e.g., `deltacommit` and `compaction`) often run concurrently. + - The CLI doesn't help correlate or visualize overlapping operations. +2. Lack of temporal context + - Timestamps are embedded in filenames but are hard to compare visually - year, month and day can be quickly + determined, but minutes and seconds are harder to parse. + - No easy way to tell how long an action took or whether it's stalling unless you manually calculate the difference + between requested and completion time. +3. Hard to spot inconsistencies or missing states + - An `inflight` compaction without a corresponding `commit` can indicate a starved/stuck compaction, which usually + blocks archiving/cleaning. + - These gaps are easy to miss when scanning filenames. + +On top of that, all timeline files are now stored as Avro binaries. Inspecting their contents requires custom Avro +readers to convert the binaries to JSON. + +## Scope + +This RFC covers visualization of metadata available in Hudi tables. All features are **READ-ONLY** - there is no support +for starting or spawning jobs that mutate a Hudi table. + +Alongside the timeline, the UI surfaces two additional read-only metadata views: the table's configuration +(`hoodie.properties`) and its schema-change history. + +The following are **out of scope**: + +- **Archived timeline:** Only the active timeline is rendered. Loading instants from LSM-based archive files is left for + future work. +- **Metadata table overlay:** The metadata table's own timeline is not shown alongside the main table timeline. +- **Write/mutation operations:** The UI cannot trigger compactions, clustering, or any write action. +- **Authentication/authorization:** No access control is added. The timeline server is assumed to run in a trusted + network, same as today. + + **Threat model:** The timeline and instant-detail views are `/v1`-parity - they read the same active-timeline and + filesystem metadata the existing `/v1/` REST APIs already serve, on the same network interface (the server binds to + all interfaces on the driver/standalone host). Two views widen the read surface beyond `/v1`, whose routes serve only + file-slice/base-file/timeline DTOs: the table-config view (`/v2/hoodie/view/table/config`) returns the full + `hoodie.properties` via `HoodieTableConfig.getProps()`, and the schema-history view + (`/v2/hoodie/view/table/schema/history`) exposes current and historical table schemas. Table properties can reference + sensitive material - KMS endpoints, lock-provider connection strings, external key/vault paths - though they rarely + embed secrets directly. The first cut serves table config unfiltered (sorted, as-is); the same content is already + readable by anyone with filesystem access to `.hoodie/hoodie.properties`. The primary control is that all UI routes, + including these two, are gated behind `--enable-ui` (off by default), with the server assumed to run on a trusted + network; a redacting/allowlisted config view is a possible future refinement for less-trusted interfaces. The UI adds + no write or mutation capability. Operators on untrusted networks should front the server with a reverse proxy or + restrict it to a private interface / localhost via network policy. + +## Implementation + +Keeping the implementation lightweight is a priority - we should add as few dependencies as possible. Changes go into +the existing `hudi-timeline-service` module, which contains a Javalin web-application that caches filesystem metadata of +a Hudi table for job executors during tagging/writing. + +The first cut runs the UI on the Timeline Server in **STANDALONE** mode (see [Configuration](#configuration)) and is +self-contained within `hudi-timeline-service`. Enabling the UI on the **EMBEDDED** timeline server inside a Spark +driver, together with a Spark UI tab, requires cross-module wiring (`hudi-client-common`, `hudi-spark-client`); it is +designed below but deferred to a follow-up to keep the initial PR small and focused. The standalone UI lands first; the +embedded/Spark linking lands next. + +The Hudi Timeline UI has two parts: the frontend and backend. + +### Architecture + +The timeline server can run standalone or embedded inside a Spark driver. In embedded mode, a tab in the Spark UI links +directly to the Hudi Timeline UI. The embedded mode and Spark UI tab (right side of the diagram below) are a planned +follow-up; the first cut is standalone-only. + +```mermaid +graph LR + Browser["Browser"] + + subgraph Driver["Standalone / Spark Driver"] + subgraph TimelineServer["Javalin (Timeline Server)"] + Static["/ui entry + /ui/static/*\n(HTML, JS, CSS)"] + API["/v2/hoodie/view/* - TimelineHandler"] + Meta["HoodieTableMetaClient\n(active timeline, config, schema)"] + + API --> Meta + end + + subgraph SparkUI["Spark UI (:4040) - embedded mode (follow-up)"] + direction TB + SparkUIPad[ ] ~~~ Tabs["[Jobs] [Stages] ... [Hudi Timeline]"] + end + + style SparkUIPad fill:none,stroke:none,color:none + + Tabs -- "link" --> Static + end + + Browser -- "HTTP" --> Static + Browser -- "HTTP" --> API + Browser -. "HTTP\n(embedded mode)" .-> SparkUI +``` + +There are two categories of requests: + +1. **Static file requests** - Javalin serves JavaScript, CSS, and library assets from the classpath + (`src/main/resources/public/`) under the `/ui/static/` URL prefix; `UiHandler` serves `index.html` at `/ui`. No + server-side rendering or template engine is needed. +2. **REST API requests** (`/v2/hoodie/view/*`) - `TimelineHandler` processes these requests, reading from a short-lived + `HoodieTableMetaClient` built for the request's basepath - its `getActiveTimeline()` for the timeline routes, and + table config/schema for the config/schema routes - and returning JSON. + +### Frontend + +The frontend is static HTML pages with vanilla JavaScript, similar to the Spark Web UI. Javalin's built-in static file +serving handles files from the classpath - no template engine (e.g., Thymeleaf) is needed and no new Java compile-time +dependencies are added. + +No frontend build pipeline (npm, webpack, vite) is needed. Contributing to the UI requires only a text editor. Three +libraries are vendored as static assets: vis-timeline (timeline rendering), Bootstrap 5 (layout/styling), and renderjson +(collapsible JSON in the detail panel). + +#### File Structure + +``` +hudi-timeline-service/src/main/resources/public/ +├── index.html # Landing page with basepath input form +├── js/ +│ └── timeline.js # vis-timeline initialization and REST API calls +├── css/ +│ └── style.css # Basic styling +└── lib/ # Vendored third-party assets (see Dependency Impact) + ├── vis-timeline/ # Timeline rendering (Apache-2.0 OR MIT) + │ ├── vis-timeline-graph2d.min.js + │ └── vis-timeline-graph2d.min.css + ├── bootstrap/ # Layout/styling (MIT) + │ ├── bootstrap.bundle.min.js + │ └── bootstrap.min.css + └── renderjson/ # Collapsible JSON detail panel (ISC) + └── renderjson.js +``` + +#### JavaScript Delivery: Bundled, No External Calls + +All three libraries are served from bundled copies under `/ui/static/lib/` (`/ui/static/lib/vis-timeline/`, +`/ui/static/lib/bootstrap/`, `/ui/static/lib/renderjson/`). The UI makes no external network calls, so it works out of +the box in air-gapped and security-conscious deployments with no extra configuration. The bundled, minified assets add +~890KB to the JAR (vis-timeline ~575KB, Bootstrap 5 ~305KB, renderjson ~11KB). + +Pinning a vendored copy (rather than loading from a CDN) keeps the UI deterministic and avoids a runtime dependency on +an external host being reachable. If automatic patch updates are wanted later, a CDN source can be added as an opt-in +config flag without changing this default. + +#### vis-timeline Configuration + +The timeline is configured with groups and items that map to Hudi's timeline model: + +- **Groups:** One row per action type - `commit`, `deltacommit`, `compaction`, `clean`, `rollback`, `clustering`, + `savepoint`, `logcompaction`, `indexing`, `restore`, `replacecommit`. These correspond to the actions in Review Comment: Confirmed, and thanks -- this was a real hole in the group model. `filterHoodieInstantsByLatestState` (`TimelineLayout:133-144`) groups on `(requestedTime, getComparableAction(action))` and keeps the highest state, and `COMPARABLE_ACTIONS` (`InstantComparatorV2:51-57`) maps `compaction -> commit`, `logcompaction -> deltacommit`, `clustering -> replacecommit`. So the `compaction`/`logcompaction`/`clustering` rows could never hold a completed instant, and the Test Plan line was indeed unsatisfiable. Rather than recovering the originating action, the UI now **follows** the mapping Hudi already applies: - **Groups are comparable actions.** `compaction` renders in the `commit` row, `logcompaction` in `deltacommit`, `clustering` in `replacecommit`. A completed compaction *is* a commit on disk and to every other component, so rendering it as one is correct, not a loss. - **Items keep their raw action** for label/colour. `InstantDTOV2` now carries both `action` and `comparableAction` (derived server-side via `getComparableAction`, so the map is not duplicated in JS): group by one, label by the other. - **Stuck-compaction diagnosis still works** -- and is arguably sharper. A pending compaction survives the filter with `action=compaction` intact, so it is an inflight `compaction` point item sitting in the `commit` row with no following range bar. That is exactly the signal. For a completed one, clicking through shows `HoodieCommitMetadata.operationType = COMPACT`. - **Test Plan restated** around the collapse: assert pending compaction maps to (`action=compaction`, `comparableAction=commit`), and that a completed compaction surfaces once as `action=commit`, rather than asking for one instant per `VALID_ACTIONS_IN_TIMELINE` entry. Please take another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
