[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Andrew Nelson added the comment: Just to mention that we have seen similar issues whilst importing scipy in a multiprocessing context. See https://github.com/scipy/scipy/issues/11479 and

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Nir Soffer added the comment: Does this really affect only python 3.7? We see this in RHEL 8.2, using python 3.6.8: https://bugzilla.redhat.com/show_bug.cgi?id=1837199#c69 Likely caused by: lvs = dict(self._lvs) Without locking. self._lvs is a dict that may contain 1000's of items.

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Change by Raymond Hettinger : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

miss-islington added the comment: New changeset 16d07812dd3833295cc001d19eea42eecbdb6ea5 by Miss Islington (bot) in branch '3.8': bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Raymond Hettinger added the comment: New changeset 75bedbe2ed4119ff18a2ea86c544b3cf08a92e75 by Raymond Hettinger in branch 'master': bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Change by miss-islington : -- nosy: +miss-islington nosy_count: 3.0 -> 4.0 pull_requests: +18964 pull_request: https://github.com/python/cpython/pull/19640 ___ Python tracker

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Change by Raymond Hettinger : -- keywords: +patch pull_requests: +18956 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19628 ___ Python tracker

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Serhiy Storchaka added the comment: I afraid that this is a part of the larger issue (see also issue31165). Seems the cause is that the GIL can be released when allocate or reallocate the memory. Does your project use some memory managing hooks? Using sys.modules.copy().items() can help a

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Raymond Hettinger added the comment: We should consider making this change just about everywhere in the standard library. Besides avoiding a race condition, running a for loop over `d.copy().items()` is more memory efficient and possibly faster than a for loop over `list(d.items())`.

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

Raymond Hettinger added the comment: Suggest making this change: -for module_name, module in list(sys.modules.items()): +for module_name, module in sys.modules.copy().items(): -- nosy: +rhettinger ___ Python tracker

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

New submission from Markus Mohrhard : We have hit an issue in the pickle module where the code throws an exception in a threaded environment: The interesting piece of the backtrace is: File "/xxx/1004060/lib/python3.7/site-packages/numpy/core/__init__.py", line 130, in _ufunc_reduce