Hello Vyas,
I will state again what I said on the issue tracker:
As for which allocator is better, there is no simple, clear-cut answer,
because it depends on the workload, the platform and the metrics you
care about (is the user more sensitive to memory footprint or execution
time?).
Sometimes people are unhappy about the behavior of the default allocator
on their workload and it turns out they can work around their issues
simply by changing the allocator, which is rather nice.
I will add that while mimalloc is maintained, it currently has a single
maintainer and he is not always active (you can easily see that from the
GitHub repository's statistics). We also have at least one recent report
of memory consumption issues with mimalloc:
https://github.com/apache/arrow/issues/47266
So for now I'm -0.5 on removing jemalloc.
Regards
Antoine.
Le 19/08/2025 à 21:22, Vyas Ramasubramani a écrit :
A few days ago I opened
https://github.com/apache/arrow/issues/47309<https://github.com/apache/arrow/issues/47309#issuecomment-3196114599>
where it was suggested that I bring this question to the mailing list:
Given that jemalloc development has
stopped<https://jasone.github.io/2025/06/12/jemalloc-postmortem/> and the repo is
archived<https://github.com/jemalloc/jemalloc> I wonder if Arrow should consider removing
that support as well. Arrow has already moved to mimalloc as the default allocator, and there are
a number of open issues that link back to jemalloc issues (I was brought here from a comment on
#44342<https://github.com/apache/arrow/issues/44342>). It might help reduce some noise on
this repo's issue tracker to remove jemalloc support altogether. I don't know if there is a
desire to support any particular system where jemalloc is still preferred, though. If so, then
delaying jemalloc support removal could make sense too. Just thought I'd raise the issue.
If developers would prefer to keep jemalloc support in Arrow until someone
finds an intractable issue to resolve, that would be fine. I have seen enough
other jemalloc-related issues on the issue board to suggest that there are
sufficient edge cases to consider remove jemalloc as an option regardless,
though. Given that jemalloc is no longer the default in Arrow, I would not be
surprised to see bit rot in jemalloc support going unnoticed for multiple
releases, and it could be worth nipping that in the bud.
