yum-yab commented on issue #49474: URL: https://github.com/apache/arrow/issues/49474#issuecomment-4090391587
Hey thanks for looking into that! I'll test it on the cluster to confirm it! I see, but I'm also quite confused. This pattern of memory allocation occurred when I was using the mimalloc pool (which was the default), where RSS mem patterns aligned with the allocated bytes reported by arrow. That was the point where I switched to `pa.set_memory_pool(pa.system_memory_pool())`. What is confusing to me is now that apparently internally `mimalloc` is used regardless (or else setting the `MIMALLOC_PURGE_DELAY` would have had no effect)? I thought setting the memory pool to system results removes this behaviour, but apparently not? Also, what seemed really weird is that the real RSS consumption vs pyarrow reported diverges so drastically for system memory pool. Thanks so much for looking into that, I'll report if it worked! Also I'd really appreciate if you @Anakin100100 (or anyone else) could explain me how that works, maybe the docs could also be updated to explain this, at least I'm really confused :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
