alamb commented on issue #9835: URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5063436791
> Yes, https://github.com/apache/arrow-rs/pull/10262 avoids intermediate vector re-allocations, but the allocator still needs to call mmap & munmap repeatedly for however many buffers we need, even if we just gave a perfectly sized buffer back. I am not sure that allocators call `mmap` and `munmap` 🤔 on all allocations (it depends on the allocator and the size of the allocation, as I recall) I wonder if instead of trying to do the buffer recycling internally, we contemplated adding a new API for the caller of the ipc writer to provide the target buffer (maybe Vec?) that would be checked before allocating a fresh one 🤔 That way it would be opt in behavior and the consumer of the buffer is probably in the best place to reuse it That being said, finding some realistic benchmark might be hard -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
