nsivabalan commented on issue #20401:
URL: https://github.com/apache/hudi/issues/20401#issuecomment-6372916387
hey @danny0405 :
I went over the sub tasks and have done some strategizing so that we can
focus on things to get a first end to end working version.
SDP on Hudi: work items, re-prioritized for the MVP
MVP: a single Hudi streaming table fed from an external source (Kafka)
through spark-pipelines. It's run repeatedly, and each run picks up only new
data as upserts (and inserts). No --full-refresh, no materialized views, no
truncate; those are documented as unsupported for now.
Two things make this a reasonable scope:
- The PoC already got this flow working on Spark 4.1 (runs 5, 7 and 8:
5000 events became 3897 rows, so upsert dedup worked). It needed a 30-line
catalog shim. Most of the MVP is moving that shim into Hudi properly.
- Leaving truncate alone is safe. Today's purge-style truncate also wipes
the sink's old batch-id watermark, so a full refresh doesn't skip data. Only a
half-landed #20378 would make it unsafe.
P0: needed for the MVP
┌─────┬──────────────────────────────────────────────────────────────────┬─────────────────────────┬─────────────────────────────────────┬─────────┐
│ ID │ Work item │
Tracker │ Why the MVP needs it │ Status │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ M1 │ HoodieCatalog overrides createTable(Column[]) │
split out of #20371 │ Without it the first run registers │ no PR │
│ │ │
│ a table with no hoodie.properties │ │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ │ │
│ SDP re-sends every property on │ │
│ M2 │ alterTable accepts unchanged SetProperty/RemoveProperty │
#20371 (only this part) │ every run, so the second run fails │ no PR │
│ │ │
│ (PoC run 2) │ │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ │ │
│ One bad batch kills the shared │ │
│ M3 │ Sink throws instead of calling System.exit(1) │
#20370 │ Connect server and the client hangs │ no PR │
│ │ │
│ forever │ │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ │ │
│ Needed to pick insert instead of │ │
│ M4 │ Pass per-flow spark.hoodie.* settings to the streaming sink │
#20403 │ the default upsert, and to set any │ no PR │
│ │ │
│ per-flow writer setting │ │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ │ │
│ Needed to prove "it works" and keep │ │
│ M5 │ Connect + spark-pipelines harness and Spark 4.1 CI lane │
#20402 │ it working. The PoC scripts are a │ no PR │
│ │ │
│ starting point. │ │
├─────┼──────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────┼─────────┤
│ │ End-to-end tests and example: Kafka → Hudi, two or more runs, │
the incremental part of │ │ not │
│ M6 │ row contents checked after each run, restart without new data │
#20406 plus the upsert │ This is the MVP's acceptance test │ started │
│ │ produces no new commit, upsert dedup and insert both covered │
part of #20408 │ │ │
└─────┴──────────────────────────────────────────────────────────────────┴─────────────────────────┴─────────────────────────────────────┴─────────┘
Dependencies:
M1 ──┐
M2 ──┼──► M6 end-to-end (run 1 needs M1; run 2+ needs M2)
M4 ──┤ (insert case needs M4; upsert works without it)
M3 ──┘ (makes failures debuggable)
M5 ─────► M6 runs on this harness
M1–M5 don't depend on each other, so they can all be built in parallel.
Only M3 and M4 edit the same file (HoodieStreamingSink /
DefaultSource.createSink), so merge one before the other.
P1: quality (not needed for MVP correctness)
┌─────┬──────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────┬───────────┐
│ ID │ Work item
│ Tracker │ Status │
├─────┼──────────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────┤
│ Q1 │ One timeline server and write client per query, instead of per
micro-batch │ #20400 → your #20412 → #20413 → #20415 │ in flight │
├─────┼──────────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────┤
│ Q2 │ Metadata table writer still starts and stops its own timeline
server on every commit │ not filed │ — │
└─────┴──────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────┴───────────┘
Q1 still matters for timing because it touches the same sink code as M3
and M4. Landing it first means M3 and M4 are built on top of it, so they don't
have to be rebased later.
P2: deferred until after the MVP (documented as unsupported)
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────┬──────────┐
│ Work item
│ Tracker │ Status │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ Timeline-preserving truncate plus checkpoint-generation sink id (must
ship together) │ #20404 → your draft │ parked │
│
│ #20378 │ │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ alterTable DeleteColumn of _hoodie_* columns and the other ColumnChanges
(only full refresh and MV runs │ rest of #20371 │ — │
│ hit these)
│ │ │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ Materialized views
│ #20405 │ — │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ saveAsTable(append) onto an existing table (MV only)
│ #20369 → danny0405's │ in │
│
│ #20423 │ flight │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ Incremental chaining from one Hudi streaming table to another
│ #20407 │ — │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ Delete markers, out-of-order and reinsert tests
│ rest of #20408 │ — │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┼──────────┤
│ Phase 2: DSv2 STREAMING_WRITE, admission control on the Hudi source,
Auto CDC/SCD, fan-in, atomic refresh │ — │ — │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────┴──────────┘
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
