FelixYBW commented on PR #4818: URL: https://github.com/apache/incubator-gluten/pull/4818#issuecomment-1984886356
Let's document the conversion clearly here. I have a impression that parquet-mr can take use of offheap memory for columnar data. If so the best case is that we can avoid any memcpy during the conversion (not considered about arrow=>velox conversion). But we will needs some work to avoid it. If parquet scan is onheap, we can't avoid the onheap->offheap copy, but then we should reuse the offheap buffer. If we can't make sure how the conversion happens, let's use memcpy as long we need, make sure it's 100% no memleak for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
