ashokkumar-allu opened a new pull request, #19299: URL: https://github.com/apache/hudi/pull/19299
### Describe the issue this Pull Request addresses This PR adds a new RFC — **RFC-108: Multi-dataset incremental reads in Hudi Streamer** — proposing native multi-table incremental reads in Hudi Streamer so that N Hudi tables can be read incrementally into a single, atomic ingestion job (one commit, one checkpoint). Related feature issue: Closes #19281 ### Summary and Changelog Hudi Streamer (DeltaStreamer) can currently read incrementally from exactly one source per job. Pipelines that need to combine incremental changes from **several Hudi tables** into one target must run one Streamer job per source and merge downstream (drift, no cross-source atomicity, extra watermark/monitoring overhead) or fall back to expensive full-snapshot reads for all but one table. This RFC proposes: - **`HoodieIncrMultiSource`** — a new source that reads N Hudi tables incrementally in a single batch with **independent per-table checkpoints** (each table advances on its own `numInstantsPerFetch` / missing-checkpoint strategy). - **Multi-dataset `InputBatch`** — additive `List<T> batches` alongside the existing `Option<T> batch`; `getBatch()` now fails loud if a multi-dataset batch would be silently narrowed, single-source callers are unaffected. - **`MultiDatasetTransformer` contract**, implemented by **`SqlFileBasedTransformer`** — merges the per-source datasets with a single SQL file referencing `<SRC_0>`, `<SRC_1>`, …; `<SRC>` is kept as an alias for the first dataset for backward compatibility; supports multi-statement SQL and CTE-style `CREATE [OR REPLACE] TEMPORARY VIEW`, with temp-view cleanup on both success and failure. - **`MultiTableCheckpointManager`** — persists per-table checkpoints as `table1=ts1,table2=ts2` in `deltastreamer.checkpoint.key`; a bare timestamp continues to be honored as a legacy single-source checkpoint, so existing jobs migrate without checkpoint surgery. - **`StreamSync` / `SourceFormatAdapter` wiring** — startup validation that `HoodieIncrMultiSource` requires a `MultiDatasetTransformer` (resolved even inside a `ChainedTransformer`) to prevent a silent-data-loss path where only the last source would be written; per-dataset error-event handling on the multi-dataset row path. Changelog: - `rfc/rfc-108/rfc-108.md` — new RFC document (Abstract, Background, Implementation, Rollout/Adoption Plan, Test Plan). - `rfc/README.md` — add RFC-108 entry with status `UNDER REVIEW`. No code changes in this PR — this is the RFC document only. ### Impact - **User-facing:** none from this PR (docs only). The proposed feature, when implemented, is opt-in via `source.class=HoodieIncrMultiSource` and does not alter behavior of existing single-source Streamer jobs, transformers, or checkpoints. API changes are additive; the checkpoint format degrades cleanly to the legacy single-timestamp form. - **Performance:** none from this PR. ### Risk Level none — RFC documentation only; no code or config changes. ### Documentation Update This PR **is** the design documentation for the proposed feature. No website updates are required until the implementation lands; user-facing configuration and the recommended OUTER JOIN / merge-payload patterns will be documented on the Hudi website when the feature is shipped. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable (n/a — RFC doc only; the Test Plan section of the RFC covers the tests that will accompany the implementation) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
