nsivabalan commented on issue #20401: URL: https://github.com/apache/hudi/issues/20401#issuecomment-6485534890
**Updated plan: four phases.** This re-splits the Phase 2/3 from my [earlier comment](https://github.com/apache/hudi/issues/20401#issuecomment-6373044659). Phase 1 is unchanged. Two changes, both about risk and ordering: - **Chaining gets its own phase.** It is the most uncertain track, and it shares no code with MVs or keyed deletes. Keeping it in Phase 2 would let it hold back two tracks that can ship sooner. - **Truncate moves to the end.** Full refresh is not a common operation. Today's purge truncate is safe for a single table, because it also clears the sink's batch-id metadata. Chaining can handle a purged upstream by failing with a clear "rebuild required" error. So a timeline-preserving truncate only improves the full-refresh experience, and it can come last. | Phase | Scope | Status | Target | |---|---|---|---| | 1 | Single streaming-table MVP | All code merged; acceptance run pending | **Oct 29** | | 2 | Materialized views + keyed flows (two parallel tracks) | Starting now, in parallel with the #20453 discussion | **Nov 5** | | 3 | Hudi → Hudi streaming-table chaining | Not started | **Nov 10** | | 4 | Timeline-preserving truncate and full refresh | Draft (#20378) | **Nov 13** | ### Phase 1: single streaming-table MVP (unchanged) Scope: an external source (Kafka or similar) feeding a Hudi streaming table, run repeatedly through `spark-pipelines`, with each run picking up only new data. Not supported in this phase: `--full-refresh`, MVs, truncate. - ✅ #20412 → #20413 → #20415 (#20400): sink timeline server, write-client lifetime, markers, schema changes. - ✅ #20426 (#20371 subset): `createTable(Column[])` and idempotent `SetProperty`/`RemoveProperty`. - ✅ #20452 (#20370): fail the query instead of exiting the JVM. - ✅ #20455 + #20454 (#20403): flow-scoped `spark.hoodie.*` settings reach the streaming sink, plus docs. - ⬜ Acceptance run (below). **Acceptance criteria.** Run on master with the #20453 harness via the real `spark-pipelines` CLI against Spark Connect 4.1.1, with results posted here: - An external source feeds a Hudi streaming table over two or more runs. Exact row contents are checked after each run, and each run reads only new data. - A restart with no new data produces **no new Hudi commit**. - Both upsert (record key + precombine, with duplicates deduplicated) and insert (set through a flow-scoped `spark.hoodie.*` setting) are covered. - A failing flow fails its query and does not bring down the Connect server. This covers the incremental part of #20406 and the upsert part of #20408. **Connect harness and CI (#20453 / #20402), not gating Phase 1.** The harness itself is ready. What is still under discussion is the CI shape: a per-PR in-JVM ScalaTest, plus this harness as a blocking nightly. That discussion can take its time. Phase 1 closes on the acceptance run, and Phase 2 starts now, in parallel. Once the CI shape is agreed, the acceptance scenario moves into CI. ### Phase 2: materialized views and keyed flows (on today's truncate) The two tracks are independent and can be owned and closed separately. **Track A: materialized views** - A1. #20423 (#20369): `saveAsTable` append onto an existing table. In review. - A2. The rest of #20371: - treat `DeleteColumn` of `_hoodie_*` columns as a no-op, or hide those columns from `Table.columns()`; - raise a clear `UnsupportedOperationException` instead of `MatchError` for the other `ColumnChange`s. - A3. #20405: MV flows and repeated refresh. Depends on A1 and A2. Each refresh is a full recompute on today's purge truncate. The timeline assertions in #20405 move to Phase 4. **Track A acceptance criteria:** - A pipeline with one or more MVs over Hudi and non-Hudi sources refreshes N ≥ 3 times through the CLI. - After each refresh, the MV's contents exactly match the defining query. - Repeated runs produce no catalog errors: the `_hoodie_*` column deletes and unchanged properties are tolerated. - `saveAsTable` append rejects mismatched columns or locations, as Spark does. **Track B: keyed upsert/delete and streaming-table hardening** - B1. The rest of #20408: - `_hoodie_is_deleted` delete markers; - out-of-order ordering values; - delete followed by reinsertion; - negative tests for missing or invalid key, partition and precombine fields. - B2. The rest of #20406: - AvailableNow completion, stop and failure all release sink resources; - reject unsupported fan-in (several flows writing one target) before any write; - failure and restart between the Hudi commit and the Spark checkpoint. - B3. Async table services across a write-client switch: - #20444 restores 1.2.0 behaviour; - #20445 retries runs that failed only because of the switch. Both are drafts, pending alignment with @yihua. - B4. Follow-up from #20400: the metadata-table writer still starts and stops its own embedded timeline server on every commit (to be filed). **Track B acceptance criteria:** - Keyed flows handle `_hoodie_is_deleted` deletes, out-of-order ordering values, and delete-then-reinsert, with exact contents checked. - Missing or invalid key, partition or precombine configs fail fast with clear errors. - Fan-in (several flows writing one target) is rejected before any write. - Stop, failure and AvailableNow completion release sink resources (timeline server, write client). - Failure between the Hudi commit and the Spark checkpoint recovers on restart, with no duplicates and no data loss. ### Phase 3: Hudi → Hudi streaming-table chaining - 3.1. #20407: the downstream streaming table reads the upstream table incrementally. - Define the starting offsets and how a restart resumes. - Fail with a clear "rebuild required" error when history has expired or the upstream table was physically recreated. Today's purge truncate on an upstream full refresh is one such case. - Document the AvailableNow backlog limitation (the Hudi source has no admission control yet). - 3.2. Deletes through a chain: decide whether the downstream sees upstream deletes (incremental vs. CDC read) and document the behaviour. Builds on Track B. **Acceptance criteria:** - A source → Hudi ST → Hudi ST chain stays exactly correct across repeated runs and a restart of each stage. The downstream reads only new upstream commits. - When upstream history has expired, or the upstream table was recreated (purge truncate / full refresh), the downstream fails with a clear "rebuild required" error and never emits silent or misleading increments. - Upstream delete behaviour through the chain is defined, tested and documented. ### Phase 4: timeline-preserving truncate and full refresh Each item depends on 4.1. - 4.1. #20404 (draft #20378): full-table `TRUNCATE` becomes one replacecommit, so the timeline, savepoints and metadata table are kept. The old purge stays behind an opt-in config. It ships with a sink checkpoint identifier scoped to each checkpoint generation. Still to do: - failure-injection tests; - a decision on migrating legacy `default_single_writer` entries; - tests for truncating an empty table and truncating twice; - truncate behaviour with a pending compaction or clustering; - docs. - 4.2. Streaming-table full refresh: an end-to-end `--full-refresh` test showing batches 0..N replay under the new identifier, with no skipped batches and history retained. - 4.3. MV refresh on the replacecommit truncate: switch Track A to it and add the timeline assertions from #20405. - 4.4. Chaining across an upstream logical reset: the downstream reader tells a truncate replacecommit apart from a clustering one, and fails with "rebuild required" instead of emitting misleading increments. Document the order for rebuilding downstream tables. **Acceptance criteria:** - `TRUNCATE TABLE` on a Hudi table writes one replacecommit. The timeline, savepoints and metadata table are kept, and the purge stays available behind an opt-in config. - `--full-refresh` on a streaming table replays batches 0..N under the new checkpoint generation, with no skipped batches and history retained. This holds even with a failure injected between the Hudi commit and the Spark checkpoint. - MV refresh uses the replacecommit truncate, and the #20405 timeline assertions pass. - A downstream chained table detects an upstream truncate replacecommit (as distinct from clustering) and fails with "rebuild required". ### Target dates - **Phase 1: Oct 29.** All Phase 1 code has landed. What remains is the acceptance run. #20453's CI shape is still under discussion, and that does not block this date. - **Phase 2: Nov 5.** This starts now, with tracks A and B in parallel. - **Phase 3: Nov 10.** - **Phase 4: Nov 13.** These targets are tighter than the earlier estimate. Taking chaining and truncate out of Phase 2 is what makes Nov 5 realistic, but it depends on prompt reviews. If a phase is at risk, let's flag it here early rather than at the deadline. ### Out of scope (later work) - DSv2 `STREAMING_WRITE` on `HoodieInternalV2Table` - Admission control and `Trigger.AvailableNow` for the Hudi stream source - Auto CDC / SCD1/2 (needs DSv2 `SupportsRowLevelOperations`) @danny0405 does this split work for you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
