yihua opened a new pull request, #20178: URL: https://github.com/apache/hudi/pull/20178
### Describe the issue this Pull Request addresses issue: #19673 Spark reads of Hudi tables run entirely on the JVM. Apache DataFusion Comet makes the operators above the scan native, and hudi-rs now has a file group reader with Java-parity merge semantics, but there is no design on record for how the two fit together with Hudi, or for how the Hudi scan itself becomes native. ### Summary and Changelog Adds RFC-113, which captures the design: `hudi-native-spark-bundle` packaging Comet next to Hudi (#19674, under review), and a native Hudi scan that reads file slices through hudi-rs inside Comet's native engine and streams Arrow batches into the DataFusion operators above it. The RFC records the split of responsibilities (planning on the JVM, reading in Rust), the Arrow C data interface crossing that decouples the two projects' Arrow and DataFusion versions, the eligibility gates and JVM fallback, the pinned Comet overlay and its drift check, and the current implementation and validation state so the work can be resumed. Number claimed in #20177; this PR lands after it. The native scan code is in #20170 (sub-task #20176). ### Impact None, documentation only. ### Risk Level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
