Neal Norwitz <[EMAIL PROTECTED]> added the comment: On Mon, Mar 17, 2008 at 11:55 AM, Alec Thomas <[EMAIL PROTECTED]> wrote: > > Alec Thomas <[EMAIL PROTECTED]> added the comment: > > Hi Neal, > > This seems to be a more general problem than just unicode.
Kinda, sorta. The general issue is the pattern of memory allocation/deallocation. In the case of >>> x = [(1, 2, 3, 4, i) for i in xrange(800000)] The memory that is not returned is in the integer free list. If this code is changed to: >>> for x in ((1, 2, 3, 4, i) for i in xrange(800000)): pass That doesn't hold on to any extra memory. The problem is that holes are left in memory and a lot of it can't always be returned to the O/S. It can still be re-used by python. > Both exhibit the same behaviour. Naively to me it seems like using the > custom allocator uniformly would fix this problem. In general, we should find places (like unicode) that use PyMem_* (ie, system malloc) and replace them with PyObject_* (pymalloc). That should improve the behaviour, but there will always be some allocation patterns that will be suboptimal. Note that using pymalloc will only help for objects < 256 bytes. Larger objects are delegated to the system malloc and will still exhibit some of the bad problems. Alec, can you find places that are using the PyMem_* interface and create a patch to fix them. That would be great! __________________________________ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2321> __________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
