[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Torben Paetz]

Change by Torben Paetz :


--
nosy: +tpaetz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Daniel Romberg]

Daniel Romberg  added the comment:

I think this issue needs to be reopened. The problem has not been solved 
completely. We experience a lot fewer crashes in weakref than before this fix, 
however, there are recursive situations in which copy() is invoked while 
iterating the WeakValueDictionary (e.g., in our case it is a signal/slot 
implementation where the slots are stored in a WeakValueDictionary). 
_commit_removals(), which is called at the beginning of the copy operation, 
might change the dictionary if there are items that are to be removed. If there 
is an ongoing iteration, the corresponding RuntimeError is raised.

I haven't thought that through entirely, but I wonder whether the copy (and 
also deepcopy) operation could just blindly copy everything without "committing 
removals". After the copy, both instances would do their _commit_removals on 
their own upon access.

--
nosy: +djromberg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 48769a28ad6ef4183508951fa6a378531ace26a4 by Antoine Pitrou (Miss 
Islington (bot)) in branch '3.7':
bpo-35615: Fix crashes when copying a Weak{Key,Value}Dictionary. (GH-11384) 
(GH-11785)
https://github.com/python/cpython/commit/48769a28ad6ef4183508951fa6a378531ace26a4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[miss-islington]

Change by miss-islington :


--
pull_requests: +11773, 11774, 11775

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 96d37dbcd23e65a7a57819aeced9034296ef747e by Antoine Pitrou (Fish) 
in branch 'master':
bpo-35615: Fix crashes when copying a Weak{Key,Value}Dictionary. (GH-11384)
https://github.com/python/cpython/commit/96d37dbcd23e65a7a57819aeced9034296ef747e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[miss-islington]

Change by miss-islington :


--
pull_requests: +11773, 11774

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[miss-islington]

Change by miss-islington :


--
pull_requests: +11773

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +fdrake, pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Fish Wang]

Fish Wang  added the comment:

Just checked weakref.py on different branches, I think this bug exists in 
Python 2.7, 3.4, 3.5, and 3.6 as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Emmanuel Arias]

Change by Emmanuel Arias :


--
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Fish Wang]

Fish Wang  added the comment:

Just submitted a PR against the master branch on GitHub.

> This can be apply to 3.8?

I think so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch, patch, patch
pull_requests: +10737, 10738, 10739, 10740
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch
pull_requests: +10737, 10738
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch, patch
pull_requests: +10737, 10738, 10739
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +10737
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Fish Wang]

Fish Wang  added the comment:

Thanks for your reply.

I'm preparing a PR. However, I'm not sure how to write a reliable test case to 
trigger the crash outside my application. I will submit the PR for now, and see 
if anyone on the mailing list has a better idea of what a reliable test case 
should look like.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Emmanuel Arias]

Emmanuel Arias  added the comment:

Hi!

Seems like a bug. Please provide a patch. 

This can be apply to 3.8?

--
nosy: +eamanu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Hi, I think this is indeed a bug, `copy()` is expected to succeed. I was able 
to reproduce the issue in an interpreter but did not succeed to write a test 
case that triggers the race-condition.

I think the fix you suggested is the right solution.

--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35615] "RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary

[Fish Wang]

New submission from Fish Wang :

I come across this issue recently when developing a multi-threaded PySide2 (Qt) 
application. When I'm calling .copy() on a WeakValueDictionary, there is a high 
chance that my application crashes with the following stack backtrace:

--

Traceback (most recent call last):
  File "F:\angr\angr-management\angrmanagement\ui\widgets\qdisasm_graph.py", 
line 239, in mouseDoubleClickEvent
block.on_mouse_doubleclicked(event.button(), 
self._to_graph_pos(event.pos()))
  File "F:\angr\angr-management\angrmanagement\ui\widgets\qblock.py", line 130, 
in on_mouse_doubleclicked
obj.on_mouse_doubleclicked(button, pos)
  File "F:\angr\angr-management\angrmanagement\ui\widgets\qinstruction.py", 
line 128, in on_mouse_doubleclicked
op.on_mouse_doubleclicked(button, pos)
  File "F:\angr\angr-management\angrmanagement\ui\widgets\qoperand.py", line 
162, in on_mouse_doubleclicked
self.disasm_view.jump_to(self._branch_target, src_ins_addr=self.insn.addr)
  File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", 
line 258, in jump_to
self._jump_to(addr)
  File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", 
line 372, in _jump_to
self._display_function(function)
  File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", 
line 343, in _display_function
vr = self.workspace.instance.project.analyses.VariableRecoveryFast(the_func)
  File "f:\angr\angr\angr\analyses\analysis.py", line 109, in __call__
oself.__init__(*args, **kwargs)
  File 
"f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 
618, in __init__
self._analyze()
  File "f:\angr\angr\angr\analyses\forward_analysis.py", line 557, in _analyze
self._analysis_core_graph()
  File "f:\angr\angr\angr\analyses\forward_analysis.py", line 580, in 
_analysis_core_graph
changed, output_state = self._run_on_node(n, job_state)
  File 
"f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 
712, in _run_on_node
input_state = prev_state.merge(input_state, successor=node.addr)
  File 
"f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 
488, in merge
merged_register_region = 
self.register_region.copy().replace(replacements).merge(other.register_region,
  File "f:\angr\angr\angr\keyed_region.py", line 159, in copy
kr._object_mapping = self._object_mapping.copy()
  File "D:\My Program Files\Python37\lib\weakref.py", line 174, in copy
for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration

--

I went ahead and read the related methods in Lib\weakref.py, and it seems to me 
that the WeakValueDictionary.copy() method is missing the protection of an 
_IterationGuard: It is iterating through self.data.items(), which, might have 
entries removed because of GC during the iteration.

It seems that this crash can be fixed by wrapping the iteration with `with 
_IterationGuard(self):`. It worked for me in my tests.

If my above analysis is correct, the following methods all require protection 
of _IterationGuard (which are currently missing):

- WeakValueDictionary.copy()
- WeakValueDictionary.__deepcopy__()
- WeakKeyDictionary.copy()
- WeakKeyDictionary.__deepcopy__()

Please let me know if this is a legitimate issue, in which case I will be happy 
to provide a patch. Thanks.

--
components: Library (Lib)
messages: 332734
nosy: Fish Wang
priority: normal
severity: normal
status: open
title: "RuntimeError: Dictionary changed size during iteration" when copying a 
WeakValueDictionary
type: crash
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com