I would recommend to use jemalloc instead. Thanks, Cheng Pan
> On Aug 25, 2026, at 12:05, 毛玉伟 <[email protected]> wrote: > > Hi community, > > In our Celeborn cluster, the MALLOC_ARENA_MAX environment variable is not > configured on the Worker machines. Without this configuration, glibc defaults > to 8 * number of CPU cores. Since our machines have 128 cores, > MALLOC_ARENA_MAX defaults to $8 \times 128 = 1024$. > > Under this setup, the memory occupied by glibc arenas in the Worker process > is extremely high. For example, arena=7.58GiB, heaps=545, and > totalRSS=23.17GiB. Furthermore, the memory consumed by these arenas is not > accounted for in Direct memory usage by the MemoryManager, leading to an > excessively high overall memory footprint for the Worker process. > > > > Question： > > Our research indicates that many big data components recommend setting > MALLOC_ARENA_MAX to 4 or 8. Does the Celeborn community have a recommended > value for this? Additionally, are there any future plans to address the issue > where Direct memory usage does not account for glibc arena memory?
