lidavidm commented on issue #13837: URL: https://github.com/apache/arrow/issues/13837#issuecomment-1210541969
The pools only track memory allocated by Arrow for the purpose of data buffers, so it would make sense that they mostly show 0. (For Flight, we try to reuse gRPC's allocations, so the memory isn't tracked by pools.) But ReleaseUnused just calls malloc_trim for you, so it works regardless of who actually allocated the memory. If releasing memory in the system pool is freeing up memory, it's quite likely gRPC's allocations (as rok has already mentioned). From what I see in https://man7.org/linux/man-pages/man3/mallopt.3.html, the trim_threshold may not have an effect if allocations are large enough that malloc uses mmap instead of sbrk. But a manual malloc_trim would still have an effect. That may explain what's going on here. If memory usage is a concern, it may be worth keeping a manual malloc_trim at the end of an RPC call/end of the transformation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
