m-v-w commented on PR #46190: URL: https://github.com/apache/arrow/pull/46190#issuecomment-2902369166
Hi @adamreeve , @CurtHagenlocher , thanks for considering my PR. The issue I encountered with `ArrowStreamWriter` is it allocates large buffers on compression and array slicing which will only be released on finalization by the GC. This caused GC pressure in my app. Not sure what a good benchmark would look like. It depends on the `MemoryAllocator` implementation. Making sure all allocations are disposed as soon as possible would allow the user to use a pooled `MemoryAllocator` implementation for instance. My implementation of the `TryCompress` method for LZ4 is bad, it wasn't a priority since I don't use it. But I'll come up with a better solution. I think having the possibility to implement a compression codec that's allocation free is useful. I'll address the raised issues and make the required changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
