tustvold commented on PR #6590: URL: https://github.com/apache/arrow-rs/pull/6590#issuecomment-2449563455
I think we would need to get this mechanism integrated into `Array` and `RecordBatch` for it to be usable by DataFusion. As for the `MutableBuffer` changes, they look reasonable, but I suspect the ROI will be much lower. Integrating it into every kernel that produces arrays is likely simply impractical, with a more pragmatic approach to simply account for the memory usage once the arrays have been constructed. DF could even choose to only pay this overhead in operators that buffer data, and accepting that short-lived arrays are not accounted for, which is what it currently does. This doesn't mean we can't add the `MutableBuffer` changes, I'm just not really sure they'd actually get used in practice, and perhaps we want to think about a concrete use-case before adding them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
