andygrove opened a new issue, #3770: URL: https://github.com/apache/datafusion-comet/issues/3770
### What is the problem the feature request solves? CometArrowConverters reuses Arrow buffers between batches for efficiency. This is used from `CometLocalTableScanExec` and `CometSparkToColumnarExec`. However, when the native `ScanExec` reads the batches, it always performs a deep copy for safety, to the re-use optimization seems to just add complexity rather than providing any performance benefit, although it is possible that the allocations in native code may be more efficient than allocations in JVM code. I think we should investigate this more and consider simplifying the approach. ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
