yihua opened a new pull request, #20170: URL: https://github.com/apache/hudi/pull/20170
### Describe the issue this Pull Request addresses issue: #19673 Comet leaves the Hudi scan on the JVM (`Unsupported file format HoodieFileGroupReaderBasedFileFormat`), so with #19674 Parquet decode stays on the JVM and a merge-on-read read pays a per-cell `CometSparkRowToColumnar` bridge before anything runs natively. RFC-113 (phase 2) covers the design. **Do not merge.** Stacked on #19674, review that first; the diff shows the parent's three commits until it merges, and the code will be squashed and re-reviewed before this leaves draft. Design: RFC-113 (Phase 2). Opened to capture the current state. ### Summary and Changelog Adds a native Hudi scan to the Comet bundle: the Hudi scan node is replaced by `CometHudiNativeScanExec`, whose native operator reads each task's file slices through hudi-rs (Parquet decode plus MOR log merge in Rust) and streams Arrow batches straight into the DataFusion operators above it. Planning stays in Hudi's existing Spark integration; the operator only receives the table base URI, the read options, the projected schema, and the slice paths. - `hudi-spark-datasource/hudi-spark-comet`: the Spark-side glue (scan provider with eligibility gates, the exec node, the per-partition plan data injector), compiled only under `-Dcomet.hudi.build`. - `packaging/hudi-native-spark-bundle/comet/`: Comet is built from a pinned commit (1.0.0) with a changed-files-only overlay that adds the `HudiScan` proto, the native operator, and a `ServiceLoader` hook for external scan providers; `build_comet.sh` verifies the overlay against the pinned upstream blob SHAs before applying it. hudi-rs is pinned by git rev to the arrow 58 / DataFusion 54.1 upgrade that shipped as 0.5.0. - Bundle validation gains native-scan probes gated on `NATIVE_HUDI_SCAN=1`. Runtime gate: `spark.comet.scan.hudi.enabled=true` (off by default). Any query shape outside the gates (incremental, CDC, bootstrap, nested types, table version outside 6/8/9, MOR below version 8, dropped partition columns, non-identity key generators with partition-column projection) leaves the plan untouched. Still owed before this is ready for review: an adversarial self-review of the authored code, CI for the Comet source build, the S3 storage option passthrough, and a decision on whether the overlay should store patches instead of full file copies. ### Impact New module and build profile only. Without `-Dcomet.hudi.build` the bundle is byte-identical to #19674. With it, the bundle depends on a locally built Comet at a distinct version (`1.0.0-hudi-<n>`), so the unmodified Central artifact can never satisfy the dependency silently. ### Risk Level low Opt-in at build time and at runtime, with the JVM scan as the fallback for every unsupported shape. Validated locally on Spark 3.5.5 (macOS arm64, Corretto 17, table version 8 tables): copy-on-write and merge-on-read snapshot rows and aggregates identical to the JVM read, no row bridge on the MOR path, time travel identical and native, and a CoW-join-MOR probe fully native with both sides through the native scan. Disabling the config leaves the JVM plan untouched, and the #19674 validation still passes without the build flag. ### Documentation Update The module README under `packaging/hudi-native-spark-bundle/comet/` documents the build order and pins. Website docs for the native bundle and the new config are needed once this leaves draft. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
