yihua opened a new issue, #20176: URL: https://github.com/apache/hudi/issues/20176
part of #19673 ### Problem With `hudi-native-spark-bundle`, Comet runs the operators above the Hudi scan natively but leaves the scan itself on the JVM: it rejects `HoodieFileGroupReaderBasedFileFormat`, so Parquet decode stays on Spark, and a merge-on-read read is bridged row by row through `CometSparkRowToColumnar`, a per-cell conversion that dominates the read. ### Proposal Add a Comet scan operator that reads Hudi file slices through hudi-rs (Parquet decode plus MOR log merge in Rust) and streams Arrow batches straight into the DataFusion operators above it. Planning stays in Hudi's existing Spark integration; the native side receives only the table base URI, read options, projected schema, and slice paths. Opt-in at build time and at runtime, with the JVM scan as the fallback for every unsupported query shape. Design in RFC-113; draft implementation in #20170. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
