[issue11849] glibc allocator doesn't release all free()ed memory

[Bob Kline]

Bob Kline  added the comment:

> ... jemalloc can reduce memory usage ...

Thanks for the tip. I downloaded the source and successfully built the DLL, 
then went looking for a way to get it loaded. Unfortunately, DLL injection, 
which is needed to use this allocator in Python, seems to be much better 
supported on Linux than on Windows. Basically, Microsoft's documentation [1] 
for AppInit_DLL, the shim for DLL injection on Windows, says (in effect) 
"here's how to use this technique, but we don't recommend using it, so here's a 
link [2] for what we recommend you do instead. That link takes you to "Try 
searching for what you need. This page doesn’t exist."

[1] 
https://support.microsoft.com/en-us/help/197571/working-with-the-appinit-dlls-registry-value
[2] https://support.microsoft.com/en-us/help/134655

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[INADA Naoki]

INADA Naoki  added the comment:

FYI, jemalloc can reduce memory usage, especially when application
is multithreaded.

https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
https://zapier.com/engineering/celery-python-jemalloc/

--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Bob Kline]

Bob Kline  added the comment:

Thanks for your responses to my comments. I'm working as hard as I can to get 
my customer's systems migrated into the Python 3 world, and I appreciate the 
efforts of the community to provide incentives (such as the resolution for this 
failure) for developers to upgrade. However, it's a delicate balancing act 
sometimes, given that we have critical places in our system for which the same 
code runs more than twice as slowly on Python 3.6 as on Python 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Well, memory fragmentation can happen with any allocation scheme, and it's 
possible even Python 3 isn't immune to this.  Backporting performance 
improvements is a strain on our resources and also constitutes a maintenance 
threat (what if the bug hides in the new code?).  And Python 2.7 is really 
nearing its end-of-life more and more everyday.  So IMHO it's a no-no.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Bob Kline]

Bob Kline  added the comment:

Sorry, I should have used the language of the patch author ("the resolution"). 
Without the resolution, Python 2.7 eventually runs out of memory and crashes 
for some correctly written user code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

It's not really a fix, it's an improvement, and as such doesn't belong in 2.7.  
Using malloc() and free() is not a bug in itself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Bob Kline]

Bob Kline  added the comment:

Would it be inappropriate for this fix to be applied to 2.7?

--
nosy: +bkline

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[STINNER Victor]

STINNER Victor added the comment:

Extract of the "workaround" section:
"You could also run your Python jobs using Jython, which uses the Java JVM
and does not exhibit this behavior. Likewise, you could upgrade to Python
3.3 ,"

Which contains a link to this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Tim Peters]

Tim Peters added the comment:

[@haypo]
> http://python.dzone.com/articles/diagnosing-memory-leaks-python
> Great job! Using mmap() for arenas is the best solution for this issue.

?  I read the article, and they stopped when they found "there seemed to be a 
ton of tiny little objects around, like integers.".  Ints aren't allocated from 
arenas to begin wtih - they have their own (immortal & unbounded) free list in 
Python2.  No change to pymalloc could make any difference to that.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[STINNER Victor]

STINNER Victor added the comment:

I just found this issue from this article:
http://python.dzone.com/articles/diagnosing-memory-leaks-python

Great job! Using mmap() for arenas is the best solution for this issue. I did 
something similar on a completly different project (also using its own 
dedicated memory allocator) for workaround the fragmentation of the heap memory.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Roundup Robot]

Roundup Robot  added the comment:

New changeset e7aa72e6aad4 by Antoine Pitrou in branch 'default':
Better resolution for issue #11849: Ensure that free()d memory arenas are 
really released
http://hg.python.org/cpython/rev/e7aa72e6aad4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Charles-François Natali  added the comment:

> Hmm, quite slow indeed, are you sure you're not running in debug mode?
>

Well, yes, but it's no faster with a non-debug build: my laptop is
really crawling :-)

> If the performance regression is limited to read(), I don't think it's
> really an issue, but using mmap/munmap explicitly would probably benicer
> anyway (1° because it lets the glibc choose whatever heuristic is best,
> 2° because it would help release memory on more systems than just glibc
> systems). I think limiting ourselves to systems which have
> MMAP_ANONYMOUS is good enough.
>

Agreed.
Here's a patch.

--
Added file: http://bugs.python.org/file23782/arenas_mmap.diff

___
Python tracker 

___diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -2,8 +2,11 @@
 
 #ifdef WITH_PYMALLOC
 
-#ifdef HAVE_MALLOPT_MMAP_THRESHOLD
-  #include 
+#ifdef HAVE_MMAP
+ #include 
+ #ifdef MAP_ANONYMOUS
+  #define ARENAS_USE_MMAP
+ #endif
 #endif
 
 #ifdef WITH_VALGRIND
@@ -183,15 +186,15 @@
 /*
  * The allocator sub-allocates  blocks of memory (called arenas) aligned
  * on a page boundary. This is a reserved virtual address space for the
- * current process (obtained through a malloc call). In no way this means
- * that the memory arenas will be used entirely. A malloc() is usually
- * an address range reservation for  bytes, unless all pages within this
- * space are referenced subsequently. So malloc'ing big blocks and not using
- * them does not mean "wasting memory". It's an addressable range wastage...
+ * current process (obtained through a malloc()/mmap() call). In no way this
+ * means that the memory arenas will be used entirely. A malloc() is
+ * usually an address range reservation for  bytes, unless all pages 
within
+ * this space are referenced subsequently. So malloc'ing big blocks and not
+ * using them does not mean "wasting memory". It's an addressable range
+ * wastage...
  *
- * Therefore, allocating arenas with malloc is not optimal, because there is
- * some address space wastage, but this is the most portable way to request
- * memory from the system across various platforms.
+ * Arenas are allocated with mmap() on systems supporting anonymous memory
+ * mappings to reduce heap fragmentation.
  */
 #define ARENA_SIZE  (256 << 10) /* 256KB */
 
@@ -557,11 +560,6 @@
 if (numarenas > PY_SIZE_MAX / sizeof(*arenas))
 return NULL;/* overflow */
 #endif
-#ifdef HAVE_MALLOPT_MMAP_THRESHOLD
-/* Ensure arenas are allocated by mmap to avoid heap fragmentation. */
-if (numarenas == INITIAL_ARENA_OBJECTS)
-mallopt(M_MMAP_THRESHOLD, ARENA_SIZE);
-#endif
 nbytes = numarenas * sizeof(*arenas);
 arenaobj = (struct arena_object *)realloc(arenas, nbytes);
 if (arenaobj == NULL)
@@ -594,7 +592,12 @@
 arenaobj = unused_arena_objects;
 unused_arena_objects = arenaobj->nextarena;
 assert(arenaobj->address == 0);
+#ifdef ARENAS_USE_MMAP
+arenaobj->address = (uptr)mmap(NULL, ARENA_SIZE, PROT_READ|PROT_WRITE,
+   MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+#else
 arenaobj->address = (uptr)malloc(ARENA_SIZE);
+#endif
 if (arenaobj->address == 0) {
 /* The allocation failed: return NULL after putting the
  * arenaobj back.
@@ -1071,7 +1074,11 @@
 unused_arena_objects = ao;
 
 /* Free the entire arena. */
+#ifdef ARENAS_USE_MMAP
+munmap((void *)ao->address, ARENA_SIZE);
+#else
 free((void *)ao->address);
+#endif
 ao->address = 0;/* mark unassociated */
 --narenas_currently_allocated;
 
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2567,8 +2567,8 @@
  getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \
  getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \
  if_nameindex \
- initgroups kill killpg lchmod lchown lockf linkat lstat lutimes memrchr \
- mbrtowc mkdirat mkfifo \
+ initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
+ memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock 
poll \
  posix_fallocate posix_fadvise pread \
  pthread_init pthread_kill putenv pwrite readlink readlinkat readv realpath 
renameat \
@@ -2679,15 +2679,6 @@
   [AC_MSG_RESULT(no)
 ])
 
-AC_MSG_CHECKING(whether mallopt can set malloc mmap threshold)
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
-#include 
-]], [[mallopt(M_MMAP_THRESHOLD, 256 * 1024)]])],
-  [AC_DEFINE(HAVE_MALLOPT_MMAP_THRESHOLD, 1, Define if mallopt can set malloc 
mmap threshold.)
-   AC_MSG_RESULT(yes)],
-  [AC_MSG_RESULT(no)
-])
-
 AC_MSG_CHECKING(for broken unsetenv)
 AC_C

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> On my box:
> default:
> $ ./python -m timeit -s "n=30; f=open('/tmp/10MB.bin', 'rb');
> b=bytearray(n)" "f.seek(0);f.readinto(b)"
> 1000 loops, best of 3: 640 usec per loop
> 
> default without patch ("$ hg revert -r 68258 Objects/obmalloc.c && make"):
> $ ./python -m timeit -s "n=30; f=open('/tmp/10MB.bin', 'rb');
> b=bytearray(n)" "f.seek(0);f.readinto(b)"
> 1000 loops, best of 3: 663 usec per loop
> 
> I'm just observing a random variance (but my computer is maybe too
> slow to notice).

Hmm, quite slow indeed, are you sure you're not running in debug mode?

> However, I really don't see how the patch could play a role here.
> 
> Concerning the slight performance regression, if it's a problem, I see
> two options:
> - revert the patch
> - replace calls to malloc()/free() by mmap()/munmap() to allocate/free
> arenas (but I'm not sure anonymous mappings are supported by every OS
> out there, so this might lead to some ugly #ifdef's...)

If the performance regression is limited to read(), I don't think it's
really an issue, but using mmap/munmap explicitly would probably benicer
anyway (1° because it lets the glibc choose whatever heuristic is best,
2° because it would help release memory on more systems than just glibc
systems). I think limiting ourselves to systems which have
MMAP_ANONYMOUS is good enough.

Here is what the glibc malloc does btw:

/*
   Nearly all versions of mmap support MAP_ANONYMOUS,
   so the following is unlikely to be needed, but is
   supplied just in case.
*/

#ifndef MAP_ANONYMOUS

static int dev_zero_fd = -1; /* Cached file descriptor for /dev/zero. */

#define MMAP(addr, size, prot, flags) ((dev_zero_fd < 0) ? \
 (dev_zero_fd = open("/dev/zero", O_RDWR), \
  mmap((addr), (size), (prot), (flags), dev_zero_fd, 0)) : \
   mmap((addr), (size), (prot), (flags), dev_zero_fd, 0))

#else

#define MMAP(addr, size, prot, flags) \
 (mmap((addr), (size), (prot), (flags)|MAP_ANONYMOUS, -1, 0))

#endif

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Charles-François Natali  added the comment:

> However, there's still another strange regression:
>
> $ ./python -m timeit \
>   -s "n=30; f=open('10MB.bin', 'rb', buffering=0); b=bytearray(n)" \
>   "f.seek(0);f.readinto(b)"
>
> -> default branch:
> 1 loops, best of 3: 43 usec per loop
> -> default branch with patch reverted:
> 1 loops, best of 3: 27.5 usec per loop
>
> FileIO.readinto executes a single read() into the passed buffer.

On my box:
default:
$ ./python -m timeit -s "n=30; f=open('/tmp/10MB.bin', 'rb');
b=bytearray(n)" "f.seek(0);f.readinto(b)"
1000 loops, best of 3: 640 usec per loop

default without patch ("$ hg revert -r 68258 Objects/obmalloc.c && make"):
$ ./python -m timeit -s "n=30; f=open('/tmp/10MB.bin', 'rb');
b=bytearray(n)" "f.seek(0);f.readinto(b)"
1000 loops, best of 3: 663 usec per loop

I'm just observing a random variance (but my computer is maybe too
slow to notice).
However, I really don't see how the patch could play a role here.

Concerning the slight performance regression, if it's a problem, I see
two options:
- revert the patch
- replace calls to malloc()/free() by mmap()/munmap() to allocate/free
arenas (but I'm not sure anonymous mappings are supported by every OS
out there, so this might lead to some ugly #ifdef's...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Ah, sorry, false alarm. "b[:] = b" actually makes a temporary copy of the 
bytearray when assigning to itself (!).

However, there's still another strange regression:

$ ./python -m timeit \
  -s "n=30; f=open('10MB.bin', 'rb', buffering=0); b=bytearray(n)" \
  "f.seek(0);f.readinto(b)"

-> default branch:
1 loops, best of 3: 43 usec per loop
-> default branch with patch reverted:
1 loops, best of 3: 27.5 usec per loop

FileIO.readinto executes a single read() into the passed buffer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I see you're comparing 3.2 and default: could you run the same
> benchmark on default with and without the patch ?

Same results:
-> default branch:
1000 loops, best of 3: 364 usec per loop
-> default branch with patch reverted:
1 loops, best of 3: 185 usec per loop

(with kernel 2.6.38.8-desktop-8.mga and glibc-2.12.1-11.2.mga1)

And I can reproduce on another machine:

-> default branch:
1000 loops, best of 3: 224 usec per loop
-> default branch with patch reverted:
1 loops, best of 3: 88 usec per loop

(Debian stable with kernel 2.6.32-5-686 and glibc 2.11.2-10)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Charles-François Natali  added the comment:

> For the record, this seems to make large allocations slower:
>
> -> with patch:
> $ ./python -m timeit "b'x'*20"
> 1 loops, best of 3: 27.2 usec per loop
>
> -> without patch:
> $ ./python -m timeit "b'x'*20"
> 10 loops, best of 3: 7.4 usec per loop
>

Yes, IIRC, I warned it could be a possible side effect: since we're
now using mmap() instead of brk() for large allocations (between 256B
and 32/64MB), it can be slower (that's the reason adaptive mmap
threadshold was introduced in the first place).

> More surprising is that, even ignoring the allocation cost, other operations 
> on the memory area seem more expensive:

Hum, this it strange.
I see you're comparing 3.2 and default: could you run the same
benchmark on default with and without the patch ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

More surprising is that, even ignoring the allocation cost, other operations on 
the memory area seem more expensive:

$ ./python -m timeit -s "b=bytearray(50)" "b[:] = b"
-> python 3.3:
1000 loops, best of 3: 367 usec per loop
-> python 3.2:
1 loops, best of 3: 185 usec per loop

(note how this is just a dump memcpy)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

For the record, this seems to make large allocations slower:

-> with patch:
$ ./python -m timeit "b'x'*20"
1 loops, best of 3: 27.2 usec per loop

-> without patch:
$ ./python -m timeit "b'x'*20"
10 loops, best of 3: 7.4 usec per loop

Not sure we should care, though. It's still very fast.
(noticed in 
http://mail.python.org/pipermail/python-dev/2011-November/114610.html )

--
nosy: +eli.bendersky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Roundup Robot]

Roundup Robot  added the comment:

New changeset f8a697bc3ca8 by Antoine Pitrou in branch 'default':
Issue #11849: Make it more likely for the system allocator to release
http://hg.python.org/cpython/rev/f8a697bc3ca8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Patch looks fine to me, thank you.

--
stage:  -> patch review
versions:  -Python 2.7, Python 3.1, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Changes by Charles-François Natali :


Removed file: http://bugs.python.org/file21858/pymem.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Changes by Charles-François Natali :


Removed file: http://bugs.python.org/file21696/gc_trim.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Charles-François Natali  added the comment:

> I guess the final patch will have to guard the mallopt() call with some 
> #ifdef?

Yes. See attached patch pymalloc_frag.diff
It's the first time I'm playing with autotools, so please review this part 
really carefully ;-)

> (also, I suppose a portable solution would have to call mmap() ourselves
> for allocation of arenas, but that would probably be a bit more involved)

Yes. But since it probably only affects glibc/eglibc malloc versions, I guess 
that target implementations are likely to provide mallopt(M_MMAP_THRESHOLD).
Also, performing an anonymous mappings varies even among Unices (the mmapmodule 
code is scary). I'm not talking about Windows, which I don't know at all.

--
Added file: http://bugs.python.org/file21864/pymalloc_frag.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

This is a very interesting patch, thank you.
I've tested it on Mandriva 64-bit and it indeed fixes the free() issue on the 
XML workload. I see no regression on pybench, stringbench or json/pickle 
benchmarks.

I guess the final patch will have to guard the mallopt() call with some #ifdef?
(also, I suppose a portable solution would have to call mmap() ourselves for 
allocation of arenas, but that would probably be a bit more involved)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-François Natali]

Charles-François Natali  added the comment:

I've had some time to look at this, and I've written a quick demo
patch that should - hopefully - fix this, and reduce memory
fragmentation.
 A little bit of background first:
 - a couple years ago (probably true when pymalloc was designed and
merged), glibc's malloc used brk for small and medium allocations, and
mmap for large allocations, to reduce memory fragmentation (also,
because of the processes' VM layout in older Linux 32-bit kernels, you
couldn't have a heap bigger than 1GB). The threshold for routing
requests to mmap was fixed, and had a default of 256KB (exactly the
size of an pymalloc arena). Thus, all arenas were allocated with mmap
 - in 2006, a patch was merged to make this mmap threshold dynamic,
see http://sources.redhat.com/ml/libc-alpha/2006-03/msg00033.html for
more details
 - as a consequence, with modern glibc/elibc versions, the first
arenas will be allocated through mmap, but as soon as one of them is
freed, subsequent arenas allocation will be allocated from the heap
through brk, and not mmap
 - imagine the following happens :
   1) program creates many objects
   2) to store those objects, many arenas are allocated from the heap
through brk
   3) program destroys all the objects created, except 1 which is in
the last allocated arena
   4) since the arena has at least one object in it, it's not
deallocated, and thus the heap doesn't shrink, and the memory usage
remains high (with a huge hole between the base of the heap and its
top)
 Note that 3) can be a single leaked reference, or just a variable
that doesn't get deallocated immediately. As an example, here's a demo
program that should exhibit this behaviour:

 """
 import sys
 import gc

 # allocate/de-allocate/re-allocate the array to make sure that arenas are
 # allocated through brk
 tab = []
 for i in range(100):
tab.append(i)
 tab = []
 for i in range(100):
tab.append(i)

 print('after allocation')
 sys.stdin.read(1)

 # allocate a dict at the top of the heap (actually it works even without) this
 a = {}

 # deallocate the big array
 del tab
 print('after deallocation')
 sys.stdin.read(1)

 # collect
 gc.collect()
 print('after collection')
 sys.stdin.read(1)
 """

 You should see that even after the big array has been deallocated and
collected, the memory usage doesn't decrease.

 Also, there's another factor coming into play, the linked list of
arenas ("arenas" variable in Object/obmalloc.c), which is expanded
when there are not enough arenas allocated: if this variable is
realloc()ed while the heap is really large and whithout hole in it, it
will be allocated from the top of the heap, and since it's not resized
when the number of used arenas goes down, it will remain at the top of
the heap and will also prevent the heap from shrinking.

 My demo patch (pymem.diff) thus does two things:
 1) use mallopt to fix the mmap threshold so that arenas are allocated
through mmap
 2) increase the maximum size of requests handled by pymalloc from
256B to 512B (as discussed above with Antoine). The reason is that if
a PyObject_Malloc request is not handled by pymalloc from an arena
(i.e. greater than 256B) and is less than the mmap threshold, then we
can't do anything if it's not freed and remains in the middle of the
heap. That's exactly what's happening in the OP case, some
dictionnaries aren't deallocated even after the collection (I couldn't
quite identify them, but there seems to be some UTF-8 codecs and other
stuff)

 To sum up, this patch increases greatly the likelihood of Python's
objects being allocated from arenas which should reduce fragmentation
(and seems to speed up certain operations quite a bit), and ensures
that arenas are allocated from mmap so that a single dangling object
doesn't prevent the heap from being trimmed.

 I've tested it on RHEL6 64-bit and Debian 32-bit, but it'd be great
if someone else could try it - and of course comment on the above
explanation/proposed solution.
Here's the result on Debian 32-bit:

Without patch:

*** Python 3.3.0 alpha
---   PID TTY  STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0  1843 pts/1S+ 0:00  1  1795  9892  7528  0.5 ./python
/home/cf/issue11849_test.py
  1  1843 pts/1S+ 0:16  1  1795 63584 60928  4.7 ./python
/home/cf/issue11849_test.py
  2  1843 pts/1S+ 0:33  1  1795 112772 109064  8.4
./python /home/cf/issue11849_test.py
  3  1843 pts/1S+ 0:50  1  1795 162140 159424 12.3
./python /home/cf/issue11849_test.py
  4  1843 pts/1S+ 1:06  1  1795 211376 207608 16.0
./python /home/cf/issue11849_test.py
END  1843 pts/1S+ 1:25  1  1795 260560 256888 19.8
./python /home/cf/issue11849_test.py
 GC  1843 pts/1S+ 1:26  1  1795 207276 204932 15.8
./python /home/cf/issue11849_test.py

With patch:

*** Python 3.3.0 alpha
---   PID TTY  STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0  1996 pts/1S+ 0:00  1  1795

[issue11849] glibc allocator doesn't release all free()ed memory

[Dave Malcolm]

Changes by Dave Malcolm :


--
nosy: +dmalcolm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-Francois Natali]

Charles-Francois Natali  added the comment:

> It isn't better.

Requests above 256B are directly handled by malloc, so MALLOC_MMAP_THRESHOLD_ 
should in fact be set to 256 (with 1024 I guess that on 64-bit every mid-sized 
dictionnary gets allocated with brk).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> > The MALLOC_MMAP_THRESHOLD improvement is less visible here:
> >
> 
> Are you running on 64-bit ?

Yes.

> If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls
> back to brk when there are too many mmap mappings).
> You could try with
> MALLOC_MMAP_THRESHOLD_=1024 MALLOC_MMAP_MAX_=16777216 ../opt/python
> issue11849_test.py

It isn't better.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Charles-Francois Natali]

Charles-Francois Natali  added the comment:

> The MALLOC_MMAP_THRESHOLD improvement is less visible here:
>

Are you running on 64-bit ?
If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls
back to brk when there are too many mmap mappings).
You could try with
MALLOC_MMAP_THRESHOLD_=1024 MALLOC_MMAP_MAX_=16777216 ../opt/python
issue11849_test.py

By the way, never do that in real life, it's a CPU and memory hog ;-)

I think the root cause is that glibc's malloc coalescing of free
chunks is called far less often than in the original ptmalloc version,
but I still have to dig some more.

>> By the way, I noticed that dictionnaries are never allocated through
>> pymalloc, since a new dictionnary takes more than 256B...
>
> On 64-bit builds indeed. pymalloc could be improved to handle allocations up
> to 512B. Want to try and write a patch?

Sure.
I'll open another issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[kaifeng]

kaifeng  added the comment:

Sorry for the later update.

Valgrind shows there is no memory leak (see attached valgrind.log).

The following code,
while True:
XML(gen_xml())
has an increasing memory usage in the first 5~8 iterations, and waves around a 
constant level afterwards.

So I guess there's a component, maybe libc, Python interpreter, 
ElementTree/pyexpat module or someone else, hold some memory until process ends.

--
Added file: http://bugs.python.org/file21770/valgrind.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> By the way, I noticed that dictionnaries are never allocated through
> pymalloc, since a new dictionnary takes more than 256B...

On 64-bit builds indeed. pymalloc could be improved to handle allocations up to 
512B. Want to try and write a patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

The MALLOC_MMAP_THRESHOLD improvement is less visible here:

$ MALLOC_MMAP_THRESHOLD_=1024 ../opt/python issue11849_test.py 
*** Python 3.3.0 alpha
--- USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
  0 antoine   7703  0.0  0.1  57756  8560 pts/2S+   01:16   0:00 
../opt/python issue11849_test.py
  1 antoine   7703 62.0  1.0 138892 86100 pts/2S+   01:16   0:01 
../opt/python issue11849_test.py
  2 antoine   7703 84.6  2.0 213580 160552 pts/2   S+   01:16   0:02 
../opt/python issue11849_test.py
  3 antoine   7703 97.0  2.9 288080 234972 pts/2   S+   01:16   0:03 
../opt/python issue11849_test.py
  4 antoine   7703 85.6  3.9 362852 309408 pts/2   S+   01:16   0:05 
../opt/python issue11849_test.py
  5 antoine   7703 93.4  4.8 437616 383844 pts/2   S+   01:16   0:06 
../opt/python issue11849_test.py
  6 antoine   7703 99.0  5.7 512380 458276 pts/2   S+   01:16   0:07 
../opt/python issue11849_test.py
  7 antoine   7703 89.6  6.7 591360 535672 pts/2   S+   01:16   0:08 
../opt/python issue11849_test.py
  8 antoine   7703 94.9  7.6 661676 607156 pts/2   S+   01:16   0:10 
../opt/python issue11849_test.py
  9 antoine   7703 95.5  8.6 740652 684556 pts/2   S+   01:16   0:11 
../opt/python issue11849_test.py
END antoine   7703 96.1  7.5 650432 597736 pts/2   S+   01:16   0:13 
../opt/python issue11849_test.py
 GC antoine   7703 97.2  6.5 570316 519228 pts/2   S+   01:16   0:13 
../opt/python issue11849_test.py
*** antoine   7703 90.8  6.5 569876 518792 pts/2   S+   01:16   0:13 
../opt/python issue11849_test.py


By the way, an easy fix is to use cElementTree instead of ElementTree. It still 
won't release all memory but it will eat a lot less of it, and be much faster 
as well.

--
title: ElementTree memory leak -> glibc allocator doesn't release all free()ed 
memory
versions: +Python 3.3 -Python 2.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com