viirya commented on issue #1035: URL: https://github.com/apache/datafusion-comet/issues/1035#issuecomment-2438715155
> 2. It has been indicated that this is being done by the JVM parquet reader reusing the buffer No, It is not JVM reusing the buffer, but the native reader reuses it. But I get your point. Let's replace it with native parquet reader. > 5. The ref counting machinery in Buffer provides the ability to register a custom Allocation > 6. When the reference count drops to zero it calls Allocation::drop > 7. This can be used to inform the JVM that the memory is no longer in use by arrow-rs and can be reclaimed / unpinned / reused > 8. Therefore by hooking up Allocation we can resolve the unsoundness of the current API and avoid future UB > 9. The way that arrow-rs hooks up the C Data Interface is an example of how this can be done Okay. I got what you said. I believe that the confusion began from https://github.com/apache/datafusion-comet/issues/1035#issuecomment-2438072519, https://github.com/apache/datafusion-comet/issues/1035#issuecomment-2438149174 and https://github.com/apache/datafusion-comet/issues/1035#issuecomment-2438232880. I thought you were suggested to provide custom Allocation to JVM and it is confused as JVM doesn't take it when importing arrays through FFI. What you are suggesting can be simplified to one sentence. Providing a custom Allocation when CometBuffer exports arrow buffer, and using it to detect if exported buffer is dropped from the importer. It sounds making sense. I will give it a try. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
