zhztheplayer commented on issue #10377: URL: https://github.com/apache/incubator-gluten/issues/10377#issuecomment-3264947741
@sezruby Thank you. Glad to hear we have people working on the Delta topic. The following is the current status from my end: I was prototyping a DV-compatible Delta reader at https://github.com/zhztheplayer/delta/tree/wip-velox-reader-2-2.4.0. It internally calls Velox4J, and is fairly ready for use. A Gluten-based version is also planned but the difficulty is how to add such code to Gluten's repository - it's modifying Delta's code. I am also working on the benchmark tools for read. So far we have TPC-DS. (write) I also had a PoC for [a writer](https://github.com/apache/incubator-gluten/pull/10216), but the code is less mature than the DV-compatible reader. Basically we had the alternative solution as we are doing for Iceberg, which is creating a pure Velox Delta Spark connector implementation with all features replicated in native, if we have enough resource and bandwidth to maintain it, but that's considerable engineering. If you were on that direction, we can have a conversation about it too. Would be good to have more information about your work if it's okay to share for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
