[issue19246] GC does not really free up memory in console
Esa Peuha added the comment: So best guess is that Microsoft's allocators have gotten fatally fragmented, but I don't know how to confirm/refute that. Let's test this in pure C. Compile and run the attached uglyhack.c on win32; if it reports something significantly less than 100%, it's probably safe to conclude that this has nothing to do with Python. -- nosy: +Esa.Peuha Added file: http://bugs.python.org/file32110/uglyhack.c ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
STINNER Victor added the comment: Python uses an allocator called pymalloc. For allocations smaller than 512 bytes, it uses arenas of 256 KB. If you allocate many small objects and later release most of them (but not all!), the memory is fragmented. For allocations larger than 512 bytes, Python falls back to malloc/free. It was discussed to replace pymalloc with Windows Low Fragmented Heap allocator. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
New submission from Пётр Дёмин:
Taken from http://stackoverflow.com/a/19287553/135079
When I consume all memory:
Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)]
on win32
Type help, copyright, credits or license for more information.
a = {}
for k in xrange(100): a['a' * k] = k
...
Traceback (most recent call last):
File stdin, line 1, in module
MemoryError
len(a)
64036
If we'll take summary keys length:
log(sum(xrange(64036)), 2)
30.93316861532543
we'll get near 32-bit integer overflow. After that done,
a = {}
will free all 2 Gb of allocated memory (as shown in Task Manager), but
executing:
for k in xrange(100): a[k] = k
Will cause:
MemoryError
And dictionary length something like:
len(a)
87382
--
components: Windows
messages: 199698
nosy: Пётр.Дёмин
priority: normal
severity: normal
status: open
title: GC does not really free up memory in console
type: resource usage
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19246
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +brian.curtin, tim.golden, tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
R. David Murray added the comment: My guess would be you are dealing with memory fragmentation issues, but I'll let someone more knowledgeable confirm that before closing the issue :) -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
Tim Peters added the comment:
Here on 32-bit Windows Vista, with Python 3:
C:\Python33python.exe
Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit
(Intel)] on win32
Type help, copyright, credits or license for more information.
a = {}
for k in range(100): a['a' * k] = k
...
Traceback (most recent call last):
File stdin, line 1, in module
MemoryError
del a
And here too Task Manager shows that Python has given back close to 2GB of
memory.
a = {}
for k in range(10): a['a' * k] = k
...
Traceback (most recent call last):
File stdin, line 1, in module
MemoryError
And here Task Manager shows that there's tons of memory still available.
sys._debugmallocstats() shows nothing odd after another a = {} - only 7
arenas are allocated, less than 2 MB.
Of course this has nothing to do with running in interactive mode. Same thing
happens in a program (catching MemoryError, etc).
So best guess is that Microsoft's allocators have gotten fatally fragmented,
but I don't know how to confirm/refute that.
It would be good to get some reports from non-Windows 32-bit boxes. If those
are fine, then we can be almost sure it's a Microsoft problem.
--
versions: +Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19246
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
Antoine Pitrou added the comment:
Works fine on a 32-bit Linux build (64-bit machine, though):
import sys
sys.maxsize
2147483647
a = {}
for k in range(100): a['a' * k] = k
...
Traceback (most recent call last):
File stdin, line 1, in module
MemoryError
del a
a = {}
for k in range(100): a[k] = k
...
Note that Linux says the process eats 4GB RAM.
--
nosy: +pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19246
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
STINNER Victor added the comment: int type of Python 2 uses an internal free list which has an unlimited size. If once you have 1 million different integers are the same time, the memory will never be released, even if the container storing all these integers is removed, because a reference is kept in the free list. This is a known issue of Python 2, solved indirectly in Python 3, because int type of Python 3 does not use a free list. The long type of Python 2 does not use a free list neither. -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19246] GC does not really free up memory in console
Tim Peters added the comment: haypo, there would only be a million ints here even if the loop had completed. That's trivial in context (maybe 14 MB for the free list in Python 2?). And note that I did my example run under Python 3. Besides, the OP and I both reported that Task Manager showed that Python did release almost all of the memory back to the OS. While the first MemoryError occurs when available memory has been truly exhausted, the second MemoryError occurs with way over a gigabyte of memory still free (according to Task Manager). Best guess is that it is indeed free, but so fragmented that MS C's allocator can't deal with it. That would not be unprecedented on Windows ;-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
