Patches item #1123430, was opened at 2005-02-15 21:27 Message generated for change (Comment added) made by illume You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Closed Resolution: Accepted Priority: 5 Submitted By: Evan Jones (vulturex) Assigned to: Tim Peters (tim_one) Summary: Python memory allocator: Free memory Initial Comment: This is the second version of my Python memory allocator patch. The first version was discussed on the python-dev mailing list here: http://mail.python.org/pipermail/python-dev/2005-January/ 051255.html This patch enables Python to actually return memory to the system. The current version's memory usage will only grow. This version maintains the same backwards compatability guarantees as the previous version: Calling PyObject_Free with a pointer that was returned by malloc() while NOT holding the GIL will work, and will not corrupt the state of the memory allocator. The patch modifies obmalloc.c. If it is accepted, other modifications to that file are required. In particular, I have not yet updated the WITH_MEMORY_LIMITS implementation, nor have I looked closely at the PYMALLOC_DEBUG code to see what changes (if any) are required. ---------------------------------------------------------------------- Comment By: Rene Dudfield (illume) Date: 2006-06-30 06:02 Message: Logged In: YES user_id=2042 I have done some more tests... and it seems that dictionaries do not release as much memory as lists do. Here is a modification of the last example posted. If you only let fill2() run almost all the memory is freed. fill2() uses lists. However if you let the others which use dicts run not all of the memory is freed. The processes are still 38MB when the data is del'ed. It is still much better than python2.4, however I think something fishy must be going on with dicts. <pre> import random import os def fill_this_one_doesnot_free_much_at_all(map): random.seed(0) for i in xrange(300000): k1 = (random.randrange(100), random.randrange(100), random.randrange(100)) k2 = random.randrange(100) if k1 in map: d = map[k1] else: d = dict() d[k2] = d.get(k2, 0) + 1 map[k1] = d def fill(map): random.seed(0) for i in xrange(3000000): map[i] = "asdf" class Obj: def __init__( self ): self.dumb = "hello" def fill2(map): a = [] for i in xrange(300000): o = Obj() a.append(o) return a if __name__ == "__main__": import time import gc os.system('ps v | grep memtest') d = dict() a = fill2(d) #a2 = fill(d) a2 = fill_this_one_doesnot_free_much_at_all(d) print "-"* 50 print "\n" * 3 os.system('ps v | grep memtest') del d del a gc.collect() time.sleep(2) for x in xrange(100000): pass print "-"* 50 print "\n" * 3 os.system('ps v | grep memtest') </pre> ---------------------------------------------------------------------- Comment By: Rene Dudfield (illume) Date: 2006-06-30 04:06 Message: Logged In: YES user_id=2042 Note, that this patch doesn't fix all memory leaks of this type. For example, this code below doesn't release all the memory after it is run. It starts at about 3MB, goes up to about 56MB and then only drops to 50MB. AFAIK restarting python processes is still needed to reduce memory usage for certain types of processes. <pre> import random import os def fill(map): random.seed(0) for i in xrange(300000): k1 = (random.randrange(100), random.randrange(100), random.randrange(100)) k2 = random.randrange(100) if k1 in map: d = map[k1] else: d = dict() d[k2] = d.get(k2, 0) + 1 map[k1] = d if __name__ == "__main__": os.system('ps v') d = dict() fill(d) print "-"* 50 print "\n" * 3 os.system('ps v') del d import gc gc.collect() print "-"* 50 print "\n" * 3 os.system('ps v') </pre> ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2006-03-16 01:17 Message: Logged In: YES user_id=31435 The tim-obmalloc branch was merged to the trunk (for Python 2.5a1) in revision 43059. Thank you again for your hard work and patience, Evan! ---------------------------------------------------------------------- Comment By: Evan Jones (vulturex) Date: 2006-02-23 14:29 Message: Logged In: YES user_id=539295 Great news! If you need any assistance, I would be more than happy to help. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2006-02-23 01:25 Message: Logged In: YES user_id=31435 The patch here is out of date, but that's OK. I created branch tim-obmalloc, with a working version of the patch, extensively reformatted to Python's C style, and with some minor changes to squash compiler warnings. I plan to finish this during PyCon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2005-09-05 22:43 Message: Logged In: YES user_id=31435 Assigned to me. ---------------------------------------------------------------------- Comment By: Evan Jones (vulturex) Date: 2005-05-10 04:31 Message: Logged In: YES user_id=539295 Whoops! I uploaded a "fixed" version a while ago, but I guess I didn't update the comments. The patch currently attached to this is the most up- to-date version. Sorry about that. ---------------------------------------------------------------------- Comment By: Evan Jones (vulturex) Date: 2005-02-19 14:07 Message: Logged In: YES user_id=539295 As per the discussion on python-dev, I've removed the concurrency hack. The routines in obmalloc.c now *must* be called while holding the GIL, even if the pointer was allocated with malloc(). I also finally fixed the PYMALLOC_DEBUG routines, so I believe this patch is now "complete." ---------------------------------------------------------------------- Comment By: Evan Jones (vulturex) Date: 2005-02-18 22:08 Message: Logged In: YES user_id=539295 Please ignore this patch for the moment: I'm in the process of making some fixes. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470 _______________________________________________ Patches mailing list [email protected] http://mail.python.org/mailman/listinfo/patches
