yihua commented on PR #20170: URL: https://github.com/apache/hudi/pull/20170#issuecomment-6476613139
That is right. Everything up to the file slices stays on the JVM: the Hudi relation lists partitions and files, prunes partitions, and the provider resolves the file slices as of the query instant and packs them into tasks. The plan that goes native carries, per task, the slice paths, the query instant, the projection and the filters. The hudi-rs side opens the base file and log files of each slice, merges them, null-fills columns added after the base file was written, and returns Arrow batches straight into Comet's operators. The illustration matches that boundary and would fit well in the RFC-133 doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
