[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: > ... jemalloc can reduce memory usage ... Thanks for the tip. I downloaded the source and successfully built the DLL, then went looking for a way to get it loaded. Unfortunately, DLL injection, which is needed to use this allocator in

[issue11849] glibc allocator doesn't release all free()ed memory

INADA Naoki added the comment: FYI, jemalloc can reduce memory usage, especially when application is multithreaded. https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html https://zapier.com/engineering/celery-python-jemalloc/ -- nosy: +inada.naoki

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Thanks for your responses to my comments. I'm working as hard as I can to get my customer's systems migrated into the Python 3 world, and I appreciate the efforts of the community to provide incentives (such as the resolution for this

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: Well, memory fragmentation can happen with any allocation scheme, and it's possible even Python 3 isn't immune to this. Backporting performance improvements is a strain on our resources and also constitutes a maintenance threat (what if the

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Sorry, I should have used the language of the patch author ("the resolution"). Without the resolution, Python 2.7 eventually runs out of memory and crashes for some correctly written user code. --

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: It's not really a fix, it's an improvement, and as such doesn't belong in 2.7. Using malloc() and free() is not a bug in itself. -- ___ Python tracker

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Would it be inappropriate for this fix to be applied to 2.7? -- nosy: +bkline ___ Python tracker ___

[issue11849] glibc allocator doesn't release all free()ed memory

STINNER Victor added the comment: Extract of the workaround section: You could also run your Python jobs using Jython, which uses the Java JVM and does not exhibit this behavior. Likewise, you could upgrade to Python 3.3 http://bugs.python.org/issue11849, Which contains a link to this issue.

[issue11849] glibc allocator doesn't release all free()ed memory

STINNER Victor added the comment: I just found this issue from this article: http://python.dzone.com/articles/diagnosing-memory-leaks-python Great job! Using mmap() for arenas is the best solution for this issue. I did something similar on a completly different project (also using its own

[issue11849] glibc allocator doesn't release all free()ed memory

Tim Peters added the comment: [@haypo] http://python.dzone.com/articles/diagnosing-memory-leaks-python Great job! Using mmap() for arenas is the best solution for this issue. ? I read the article, and they stopped when they found there seemed to be a ton of tiny little objects around, like

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali neolo...@free.fr added the comment: For the record, this seems to make large allocations slower: - with patch: $ ./python -m timeit b'x'*20 1 loops, best of 3: 27.2 usec per loop - without patch: $ ./python -m timeit b'x'*20 10 loops, best of 3:

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: I see you're comparing 3.2 and default: could you run the same benchmark on default with and without the patch ? Same results: - default branch: 1000 loops, best of 3: 364 usec per loop - default branch with patch reverted: 1 loops, best of

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: Ah, sorry, false alarm. b[:] = b actually makes a temporary copy of the bytearray when assigning to itself (!). However, there's still another strange regression: $ ./python -m timeit \ -s n=30; f=open('10MB.bin', 'rb', buffering=0);

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali neolo...@free.fr added the comment: However, there's still another strange regression: $ ./python -m timeit \ -s n=30; f=open('10MB.bin', 'rb', buffering=0); b=bytearray(n) \ f.seek(0);f.readinto(b) - default branch: 1 loops, best of 3: 43 usec per

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: On my box: default: $ ./python -m timeit -s n=30; f=open('/tmp/10MB.bin', 'rb'); b=bytearray(n) f.seek(0);f.readinto(b) 1000 loops, best of 3: 640 usec per loop default without patch ($ hg revert -r 68258 Objects/obmalloc.c make): $

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali neolo...@free.fr added the comment: Hmm, quite slow indeed, are you sure you're not running in debug mode? Well, yes, but it's no faster with a non-debug build: my laptop is really crawling :-) If the performance regression is limited to read(), I don't think it's

[issue11849] glibc allocator doesn't release all free()ed memory

Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset e7aa72e6aad4 by Antoine Pitrou in branch 'default': Better resolution for issue #11849: Ensure that free()d memory arenas are really released http://hg.python.org/cpython/rev/e7aa72e6aad4 --

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: For the record, this seems to make large allocations slower: - with patch: $ ./python -m timeit b'x'*20 1 loops, best of 3: 27.2 usec per loop - without patch: $ ./python -m timeit b'x'*20 10 loops, best of 3: 7.4 usec per loop

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: More surprising is that, even ignoring the allocation cost, other operations on the memory area seem more expensive: $ ./python -m timeit -s b=bytearray(50) b[:] = b - python 3.3: 1000 loops, best of 3: 367 usec per loop - python 3.2: 1

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: Patch looks fine to me, thank you. -- stage: - patch review versions: -Python 2.7, Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11849

[issue11849] glibc allocator doesn't release all free()ed memory

Roundup Robot devnull@devnull added the comment: New changeset f8a697bc3ca8 by Antoine Pitrou in branch 'default': Issue #11849: Make it more likely for the system allocator to release http://hg.python.org/cpython/rev/f8a697bc3ca8 -- nosy: +python-dev

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Antoine Pitrou pit...@free.fr: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11849 ___

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali neolo...@free.fr added the comment: I've had some time to look at this, and I've written a quick demo patch that should - hopefully - fix this, and reduce memory fragmentation. A little bit of background first: - a couple years ago (probably true when pymalloc was

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: This is a very interesting patch, thank you. I've tested it on Mandriva 64-bit and it indeed fixes the free() issue on the XML workload. I see no regression on pybench, stringbench or json/pickle benchmarks. I guess the final patch will have to

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali neolo...@free.fr added the comment: I guess the final patch will have to guard the mallopt() call with some #ifdef? Yes. See attached patch pymalloc_frag.diff It's the first time I'm playing with autotools, so please review this part really carefully ;-) (also, I

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file21696/gc_trim.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11849 ___

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file21858/pymem.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11849 ___

[issue11849] glibc allocator doesn't release all free()ed memory

kaifeng cafe...@gmail.com added the comment: Sorry for the later update. Valgrind shows there is no memory leak (see attached valgrind.log). The following code, while True: XML(gen_xml()) has an increasing memory usage in the first 5~8 iterations, and waves around a constant level

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-Francois Natali neolo...@free.fr added the comment: The MALLOC_MMAP_THRESHOLD improvement is less visible here: Are you running on 64-bit ? If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls back to brk when there are too many mmap mappings). You could try with

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: The MALLOC_MMAP_THRESHOLD improvement is less visible here: Are you running on 64-bit ? Yes. If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls back to brk when there are too many mmap mappings). You could try with

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-Francois Natali neolo...@free.fr added the comment: It isn't better. Requests above 256B are directly handled by malloc, so MALLOC_MMAP_THRESHOLD_ should in fact be set to 256 (with 1024 I guess that on 64-bit every mid-sized dictionnary gets allocated with brk). --

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Dave Malcolm dmalc...@redhat.com: -- nosy: +dmalcolm ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11849 ___ ___ Python-bugs-list

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: The MALLOC_MMAP_THRESHOLD improvement is less visible here: $ MALLOC_MMAP_THRESHOLD_=1024 ../opt/python issue11849_test.py *** Python 3.3.0 alpha --- USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND 0 antoine 7703

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou pit...@free.fr added the comment: By the way, I noticed that dictionnaries are never allocated through pymalloc, since a new dictionnary takes more than 256B... On 64-bit builds indeed. pymalloc could be improved to handle allocations up to 512B. Want to try and write a patch?