[issue6695] PyXXX_ClearFreeList for dict, set, and list

Antoine Pitrou added the comment: The clearing the dict and list freelists has been added independently, so I committed the part of the patch pertaining to sets. Thank you! -- resolution: -> fixed stage: -> committed/rejected status: open -> closed versions: +Python 3.3 -Python 3.2

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Roundup Robot added the comment: New changeset 57f0af61da53 by Antoine Pitrou in branch 'default': Issue #6695: Full garbage collection runs now clear the freelist of set objects. http://hg.python.org/cpython/rev/57f0af61da53 -- nosy: +python-dev ___

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: Patch against current tip attached. I can no longer reproduce the large memory leaks with the current tip (which is of course wonderful!), so I guess the second part of the patch (freeing the freelists during gc.collect) makes no longer sense. --

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: I uploaded updates of the three relevant patches against the current revision of the py3k branch, as the old patches no longer applied cleanly due to whitespace changes. To summarize: * The first patch, py3k-rev81387-clearfreelist-dict_set_list.patch, sim

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by Matthias Troffaes : Added file: http://bugs.python.org/file17429/py3k-rev81387-clearfreelist-time_gc_collect.patch ___ Python tracker ___ _

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by Matthias Troffaes : Added file: http://bugs.python.org/file17428/py3k-rev81387-clearfreelist-gc_collect.patch ___ Python tracker ___ __

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by Matthias Troffaes : Added file: http://bugs.python.org/file17427/py3k-rev81387-clearfreelist-dict_set_list.patch ___ Python tracker ___ ___

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by Skip Montanaro : -- nosy: -skip.montanaro ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: To aid the discussion, I attach another quick patch which reports the time spent on PyXXX_ClearFreeList calls during highest generation garbage collection (including gc.collect()). For simplicity, the timer uses clock() so the resolution is quite limited (app

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Gregory P. Smith added the comment: +1 on the PyXXX_ClearFreeList patch and calling them from gc.collect() as is done with the others. I agree with Guido, don't add a tp_free_list slot as the common case would be NULL. Regarding gc clearing freelists: I agree with Antoine and Martin. Clearing

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Martin v. Löwis added the comment: I still stand by my suggestion to free memory as a side effect of garbage collection. It may well be that an application will start re-allocating blocks that soon end up in the free list again. OTOH, it may also be that releasing those free lists, along with th

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Changes by Raymond Hettinger : -- resolution: accepted -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue6695] PyXXX_ClearFreeList for dict, set, and list

R. David Murray added the comment: FWIW, I agree with Antoine here. I think user expectation is that when "garbage" is collected, at least some freed memory will be returned to the operating system. The normal user's conception of what "garbage" is has nothing to do with cycles. It just so ha

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Antoine Pitrou added the comment: The change was originally made in r60797, « Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation ». -- nosy: +christian.heimes, loewis ___ Python tracker

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Antoine Pitrou added the comment: > When did that happen? I agree with Raymond. The cyclic gc should just > reclaim cycles. People don't care about referential cycles, they care about freeing some memory (if memory was available in infinite quantities nobody would care about breaking cycles).

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Skip Montanaro added the comment: >> I thought GC was expected to eliminate reference cycles. Antoine> Of course, but it's also the de facto API when wanting to Antoine> reclaim memory. When did that happen? I agree with Raymond. The cyclic gc should just reclaim cycles. Skip -

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Antoine Pitrou added the comment: Le samedi 15 août 2009 à 22:06 +, Raymond Hettinger a écrit : > Raymond Hettinger added the comment: > > > The reason is that users expect gc.collect() to make > > its best to diminish memory use. > > I thought GC was expected to eliminate reference cycl

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Raymond Hettinger added the comment: > The reason is that users expect gc.collect() to make > its best to diminish memory use. I thought GC was expected to eliminate reference cycles. Perhaps there ought to be a separate API, such as sys.clear_freelists(), to eliminate other memory use when

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Antoine Pitrou added the comment: The reason is that users expect gc.collect() to make its best to diminish memory use. Clearing free lists can allow deallocting some arenas which otherwise would still contain some used memory blocks. As the comment says: /* Clear all free lists * All free lis

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Raymond Hettinger added the comment: Does anyone here know why GC calls the free_xxx functions? ISTM, they cannot be involved in cycles. Free lists are kept by container objects to speed-up allocation. Having GC call the free_xxx just slows down the GC process and all the subsequent set/list/

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Raymond Hettinger added the comment: Marking the PyXXX_ClearFreeList version as approved. -- resolution: -> accepted ___ Python tracker ___ _

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Guido van Rossum added the comment: A slot in every type object for this purpose seems wasteful; the large majority of types won't have a free list. (Remember, each user-defined class allocates a full type structure on the heap.) -- assignee: gvanrossum -> ___

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Raymond Hettinger added the comment: Guido, do you want a slot assigned for this? -- assignee: -> gvanrossum nosy: +gvanrossum, rhettinger ___ Python tracker ___ ___

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: Thanks for the feedback! Attaching a new patch which implements tp_free_list slot as suggested - I hope I did it correctly. I've only implemented the new slot for dict so far, but I'm happy to tp_free_list-ify the other freelist types as well, in a future pa

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Skip Montanaro added the comment: Instead of expanding the C API for each type which supports a free list perhaps there should be a single call, say, PyObject_ClearFreeList, which takes a pointer to the appropriate type object as an argument. PyTypeObject can then grow a tp_free_list slot throug

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: I'm also attaching a test script to check the effect of the two patches on gc.collect(). If many objects are allocated, space savings appear to be relevant. Before applying the patch (debug build on linux 64 bit): Memory used (begin): 121Mb memtest 2

[issue6695] PyXXX_ClearFreeList for dict, set, and list

Matthias Troffaes added the comment: I attach a second patch which also calls the new PyXXX_ClearFreeList functions on garbage collection, during gc.collect(). -- Added file: http://bugs.python.org/file14709/py3k-clearfreelist-gc_collect.patch ___ Py

[issue6695] PyXXX_ClearFreeList for dict, set, and list

New submission from Matthias Troffaes : The Python C API provides PyXXX_ClearFreeList functions to allow the float, int, etc... freelists to be freed, potentially releasing memory to the OS earlier. Currently, there is no such API for the dict, set, and list freelists. The attached patch adds Py