2010YOUY01 commented on issue #15939: URL: https://github.com/apache/datafusion/issues/15939#issuecomment-2853267673
> This might be due to the system allocator you are using (e.g. `jemalloc` for example). Often as a performance optimization such allocators will not return memory to the underlying operating system but rather keep it in allocator specific free lists > > Maybe you can run some sort of heap profiling tool to see if you can see where the memory is being held and what allocations are still allocated (I have heard good things about `heaptrack` for example, and I have also used the `Instruments` tool on MacOSX to track allocations)( Thank you for the great insights! That was exactly the issue — I used `Instruments` and confirmed that it was `mimalloc` holding onto the memory without returning it to the OS. Manually switching to a different allocator resolved the problem. I suppose the proper fix would be to force `mimalloc` to return memory to the OS after each query finishes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
