danny0405 opened a new issue, #20401: URL: https://github.com/apache/hudi/issues/20401
### Feature Description Implement the initial Hudi source/target integration for Spark Declarative Pipelines (SDP), tracked by RFC-112 / #19997. This issue organizes contributor-sized tasks and existing prerequisite fixes; the RFC remains under review and these tasks do not imply design approval. The initial outcomes are: - Materialized-view flows write Hudi tables and refresh with defined overwrite semantics. - External sources write append streaming tables (STs) through HoodieStreamingSink, with basic upstream Hudi ST → downstream ST incremental chaining. - Flows express record-key + precombine upserts and boolean delete markers, with functional tests for deduplication, updates, ordering, and deletes. ### User Experience Users declare Hudi datasets in SDP SQL/Python and run the real spark-pipelines CLI against a configured Spark Connect server. SDP owns dependencies, execution, and checkpoints; Hudi retains commit and record-merge semantics. Initial scope is one writer per target using the existing V1 fallback. Fan-in, complete DSv2 support, Auto CDC/SCD1/2, and atomic refresh across truncate–alter–append are follow-up work. ### Hudi RFC Requirements RFC PR: https://github.com/apache/hudi/pull/19997 This is implementation tracking for an existing RFC, not a new design proposal. Shared catalog, truncate, streaming-configuration, and resource-lifecycle behavior changes require regression coverage and documentation. No new storage format is proposed. ### Contribution plan Sub-issues will be linked below. Each task should include focused tests and an implementation PR referencing its issue. Start with the Connect harness and independent prerequisite fixes; complete end-to-end acceptance after their dependencies land. Existing prerequisites: - [ ] #20369 — existing-table append routing - [ ] #20370 — propagate sink failure without JVM exit - [ ] #20371 — catalog creation and idempotent reconciliation - [ ] #20400 — query-scoped timeline-service/write-client lifetime ### Completion criteria All initial-flow tasks and prerequisites are complete, the real CLI/Connect acceptance suite passes on the supported Spark 4.1 configuration, and runnable examples document configuration ownership, restart/full-refresh semantics, and operational limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
