sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5376691586
@schenksj Let’s see how it goes. For now, I see the `delta-spark`-based implementation as the most practical approach: it builds on mature Delta planning while allowing Comet to reuse its optimized native Parquet reader. Longer term, I’m also excited about `delta-kernel-rs` as a shared foundation for native Delta integrations, and I've also heard that the Delta community is also converging on the Rust implementation. In terms of your concern, I think we should aim to keep the user-facing configuration simple, perhaps with one flag to enable Delta scans and another to opt into an experimental Rust-kernel-backed path. Ideally, both approaches would share as much integration and testing infrastructure as possible. Really appreciate all your work on this! We’re planning to move quickly with the current `delta-spark` integration and evaluate it against some very large-scale production workloads. We also plan to evaluate the `delta-kernel-rs`-based approach in the future, and I’d love to collaborate on your series and take on some work to move the Rust-based reader forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
