On 2013-06-26 13:46, Bjørn-Helge Mevik wrote: > > Hongjia Cao <[email protected]> writes: > >> I have encountered that slurmctld uses more than 20GB of virtual memory. >> But the RSS is less than 1GB. I am not sure whether this is OK or there >> is some leakage. > > On Linux boxes with newer versions of glibc, slurmctld (as well as any > other process that uses a lot of threads) will "use" a lot of VMEM. > There was a change in glibc 2.something (I think it was) in how VMEM is > allocated for threads. For instance, our slurmctld right now "uses" 16 > GiB VMEM, but only 117 MiB RSS.
Yes, this is what we're seeing as well. 6.5 GiB VMEM, 376 MB RSS. The change was that as of glibc 2.10 a more scalable malloc() implementation is used. The new implementation creates up to 8 (2 on 32-bit) pools per core, each 64 MB in size. Thus in our case, where slurmctld runs on a machine with 12 cores, we have up to 12*8*64=6144 MB in those malloc pools. See http://udrepper.livejournal.com/20948.html -- Janne Blomqvist
