[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Change by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Change by Daniel Fortunov : -- nosy: +dfortunov nosy_count: 15.0 -> 16.0 pull_requests: +19902 pull_request: https://github.com/python/cpython/pull/20687 ___ Python tracker ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Charles-François Natali added the comment: Shouldn't the documentation be updated? https://docs.python.org/3.6/library/weakref.html#weakref.WeakKeyDictionary Note Caution: Because a WeakKeyDictionary is built on top of a Python dictionary, it must not change size when iterating over it. This

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: Since this is backported, shouldn't it be closed? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___ ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Mark Lawrence breamore...@yahoo.co.uk: -- nosy: -BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___ ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Roundup Robot added the comment: New changeset 03fcc12282fc by Kristján Valur Jónsson in branch '2.7': Issue #7105: weak dict iterators are fragile because of unpredictable GC runs http://hg.python.org/cpython/rev/03fcc12282fc -- nosy: +python-dev

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: That's the spirit, Guido :) I just think people are being extra careful after the regression introduced in 2.7.5. However, IMHO we must never let the odd mistake scare us from making necessary moves. Unless Antoine explicitly objects, I think I'll

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: Strictly speaking b) is not a semantic change. Depending on your semantic definition of semantics. At any rate it is even less so than a) since the temporary list is hidden from view and the only side effect is additional memory usage. --

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: d), We could also simply issue a (documentation) warning, that the iterator methods of these dictionares are known to be fragile, and recommend that people use the keys(), values() and items() instead. --

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: d) sounds like a good enough resolution at this point. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Guido van Rossum added the comment: I'm not sure I understand the hesitation about backporting the Python 3 solution. We're acknowledging it's a bug, so the fix is not a feature. The Python 3 solution is the future. So why not fix it? -- ___ Python

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: Yes, the old memory argument. But is it valid? Is there a conceivable application where a dict of weak references would be storing a large chunk of the application memory? Remember, all of the data must be referred to from elsewhere, or else, the weak

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: Anyway, it's not for me to decide. We have currently three options: a) my first patch, which is a duplication of the 3.x work but is non-trivial and could bring stability issues b) my second patch, which will increase memory use, but to no more than

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Guido van Rossum added the comment: I'm with Antoine. Have we heard of any problems with the 3.x version of the patch? How different is it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: Here's a different approach. Simply avoid the use of iterators over the underlying container. Instead, we iterate over lists of items/keys/values etc. -- Added file: http://bugs.python.org/file32932/weakref.patch

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Guido van Rossum added the comment: I appreciate the simplicity, but I don't think it is acceptable -- if the dict is large, materializing the entire list of keys/values/items might allocate a prohibitive amount of memory. It's worse if you have code that is expected to break out of the loop.

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: No matter how it sounds, it certainly looks cleaner in code. Look at all this code, designed to work around an unexpected GC collection with various pointy bits and edge cases and special corners. Compare to explicitly just asking GC to relent, for a

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: No matter how it sounds, it certainly looks cleaner in code. It's also unsafe and invasive, since it's a process-wide setting. An iterator can be long-lived if it's being consumed slowly, so you've disabled garbage collection for an unknown amount of time,

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: Yes, the long iterator scenario is the reason it is not ideal for this scenario. The other one (gc.collect()) is easily solved by implementing this construct natively. It can be done rather simply by adding an overriding pause property to gc, with

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: Speaking of which, this is not only about GC runs, although it is the most annoying scenario (since you basically don't control when it happens). Regular resource reaping because of reference counting can also wreak havoc. About the patch, I don't know. It

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: The changes for the docs are just a port of the original patch. And indeed, these functions don't return an iterator, but a generator object. I admit I'm confused by the difference, since next() can be called directly on the generator. Still, a lot

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

STINNER Victor added the comment: What is the status of this issue? Does anyone still want to backport the fix to Python 2.7? (I found this issue while searching for test_multiprocessing failures, and I found #7060 which refers this issue.) -- nosy: +haypo

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Kristján Valur Jónsson added the comment: Attached is a version for 2.7 Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to temporarily disable GC with a context manager. -- nosy: +kristjan.jonsson Added file: http://bugs.python.org/file32689/issue7105.patch

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou added the comment: Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to temporarily disable GC with a context manager Disabling the GC in a library function sounds very ugly to me. -- ___ Python tracker

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Matthew Schwickerath matth...@wilkeglobal.com added the comment: Any plans on actually patching this in 2.7 any time soon? This is affecting our software and hanging it on random occasions. -- nosy: +qelan ___ Python tracker rep...@bugs.python.org

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Mark Dickinson dicki...@gmail.com added the comment: The issue still exists in 2.6 and 2.7. Closing issue 839159 as a duplicate of this one. -- nosy: +mark.dickinson resolution: fixed - stage: committed/rejected - status: closed - open versions: +Python 2.6, Python 2.7 -Python 3.1,

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Mark Dickinson dicki...@gmail.com: -- nosy: +BreamoreBoy, ajaksu2, dcjim, elachuni, tseaver, vdupras ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou pit...@free.fr added the comment: Committed in r77365 (py3k) and r77366 (3.1). Thank you. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Benjamin Peterson benja...@python.org added the comment: LGTM -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file15103/weakiter.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file15112/weakiter2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou pit...@free.fr added the comment: It occurs weaksets have the same problem, here is a new patch fixing them as well. -- Added file: http://bugs.python.org/file15114/weakiter3.patch ___ Python tracker rep...@bugs.python.org

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Antoine Pitrou pit...@free.fr added the comment: This new patch makes it possible to mutate the dict without messing with the delayed removal when an iterator exists. -- Added file: http://bugs.python.org/file15112/weakiter2.patch ___ Python tracker

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

New submission from Antoine Pitrou pit...@free.fr: As mentioned in issue7060, weak dict iterators are easily broken by cyclic garbage collection changing the size of the underlying dict storage: File /home/rdmurray/python/py3k/Lib/weakref.py, line 121, in items for wr in

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Changes by Jon Parise j...@indelible.org: -- nosy: +jon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___ ___ Python-bugs-list mailing list

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

Guido van Rossum gu...@python.org added the comment: delay all removals until all iterators are done +1 -- nosy: +gvanrossum ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7105 ___