whitespacer <sila...@gmail.com> added the comment:
pitrou, thanks for great diagnosing program! You are right that there is no
memory leak.
We have investigated this issue a little bit more - it looks real reason for
large memory consuming in the end is not fragmentation, just glibc doesn't
release all free memory.
So what happens in this example (according to
http://man7.org/linux/man-pages/man3/mallopt.3.html):
1) More threads leads to creating of more arenas (e.g. to more sub-heaps). It
can be tested with setting environment variable MALLOC_ARENA_MAX to 1 - memory
consumption in example will be reduced significantly. But this can be used in
real code, performance will degrade.
2) Heap trim threshold (M_TRIM_THRESHOLD) is calculated dynamically. Creation
and deletion of small arrays leads to
M_MMAP_THRESHOLD=2*4*1024*1024*sizeof(long) on 64 bit systems. So each sub-heap
in each arena can have up to 64 mb (if long is 8 bytes) of untrimmed space.
It can be tested with setting environment variable MALLOC_TRIM_THRESHOLD_ to
128*1024 - memory consumption in example will be reduced significantly because
dynamic calculation of trim threshold will be turned off. Again I doubt this
can be used in real code.
3) Maxiumum number of arenas vary on number of cpu's. On my system it looks
like it is set to 16. So I get around 16*64 of untrimmed space.
To conclude:
There is similar bug in python tracker: https://bugs.python.org/issue11849.
Message https://bugs.python.org/msg134992 states that nothing can be done in
some cases (like in attached example).
One possible solution is to use jemalloc - we have tested that it doesn't have
such problems. Limiting maximum number of arenas also helps, but can degrade
performance.
--
nosy: +whitespacer
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33444>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com