FelixYBW commented on issue #12263: URL: https://github.com/apache/gluten/issues/12263#issuecomment-4683515922
Let's reuse lance-spark and not get any velox changes involved. The perf diff from Phase 2 and 3 should be small, but the work is not trivial. Especially considering the Velox community's crazy review process. Lance decoder's memory allocation from Spark should be handled by lance-spark. lance-spark needs to implement the C-data release callback function which Gluten will use to free memory. We have such code in Gluten. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
