[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: Ok, I *think* I tracked it down to Connections getting cleaned up in a non-orderly fashion, sometimes while _run_finalizers is executing. The following patch to lib/python3.4/multiprocessing/connection.py cures the problem on my machine. @@ -124,10 +124,10 @@ self._handle = handle self._readable = readable self._writable = writable - +self._shutdown = util.Finalize(self, self._finalizer) # XXX should we use util.Finalize instead of a __del__? - -def __del__(self): + +def _finalizer(self): if self._handle is not None: self._close() -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: 20,000+ iterations of the test suite with my homebuilt 3.4.3 throw no exceptions. Weirder and weirder. Any suggestions? Without any actual knowledge (unencumbered by the thought process), a concurrency issue in the interpreter itself is my guess. Especially since I can't reproduce it at all on other versions. I'll see if I can come up with a shorter, self-contained test that can illustrate the issue too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: Finally able to repro on my old Mac Pro at work (Universal 64/32bit 3.4.3 from python.org, OS X 10.10.3). For some reason I can't quite fathom, there were two exceptions this time. Error in atexit._run_exitfuncs: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py, line 286, in _exit_function _run_finalizers(0) File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py, line 246, in _run_finalizers items = [x for x in list(_finalizer_registry.items()) if f(x)] RuntimeError: dictionary changed size during iteration Exception ignored in: Finalize object, dead Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py, line 185, in __call__ File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/queues.py, line 207, in _finalize_close File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py, line 342, in notify TypeError: 'NoneType' object is not callable -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: Ok, sorry, one more important thing I omitted in the previous comment. ^^^ The second exception above seems to be related to a new test I added, one that includes a daemonic process which itself relies on daemonic threads. The cleanup of that seems to be FUBARed, and it makes the original exception much more likely to pop up as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: More data: I've been unable to replicate this behavior on Ubuntu 15.04 on Python 3.4.3 over ~20 thousand test runs. (Same machine, running in a VM) An overnight repeated run on the original machine on OS X, 3.4.3, official distribution gave an actual frequency of 0.1% (7 crashes in 6980 runs). Approximately 3,300 runs on Python 2.7.9 on the same machine have not resulted in any crashes. I now built 3.4.3 from source using the latest clang and am trying it again, on the same machine, instead of using the official distribution. So far, 0 crashes in approximately 3,300 runs so far. At the moment, then, the problem seems localized to the official 3.4.3 binaries for OS X. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: The failing Python: 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] The non-failing Python: 3.4.3 (default, Jun 23 2015, 06:33:02) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] The non-failing python, I just downloaded the tarball and ran ./configure --prefix=/opt/python343, make, sudo make install, and then created a virtualenv using that binary. Given what you're telling me, I'll run some more loops with the non-failing python. I'm grateful you've been able to reproduce the problem, FWIW. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
Jorge Herskovic added the comment: This happens reliably between 1-5% of the time on my home Mac (a 4.0 GHz i7). My work Mac Pro, a lot slower, doesn't exhibit this behavior on the same 3.4.3 interpreter. Could it be related to a concurrency issue in the interpreter? We're attempting to replicate on other systems. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24484] multiprocessing cleanup occasionally throws exception
New submission from Jorge Herskovic: I'm writing a library on top of multiprocessing. As part of the test suite, I create and destroy dozens of processes repeatedly. About once in 50 runs, the tests complete successfully but the program crashes with: Error in atexit._run_exitfuncs: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py, line 286, in _exit_function _run_finalizers(0) File /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py, line 246, in _run_finalizers items = [x for x in list(_finalizer_registry.items()) if f(x)] RuntimeError: dictionary changed size during iteration (this is on OS X 10.10.3, Python 3.4.3; I've been unable to replicate the behavior on Python 2.7.9, 3.5.0b2, or pypy despite hundreds of runs) The problematic code is available at https://github.com/jherskovic/mpetl and the tests should be run with nosetests -- components: Library (Lib) messages: 245597 nosy: Jorge Herskovic priority: normal severity: normal status: open title: multiprocessing cleanup occasionally throws exception type: crash versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24484 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5754] Shelve module writeback parameter does not act as advertised
New submission from Jorge Herskovic jorge.r.hersko...@uth.tmc.edu: The shelve module documentation states that by default, mutations to persistent-dictionary mutable entries are not automatically written back. If the optional writeback parameter is set to True, all entries accessed are cached in memory, and written back at close time... however the implementation's __setitem__ is the following: def __setitem__(self, key, value): if self.writeback: self.cache[key] = value f = StringIO() p = Pickler(f, self._protocol) p.dump(value) self.dict[key] = f.getvalue() which maintains the cache correctly but writes back to the disk on every operation, violating the writeback documentation. Changing it to def __setitem__(self, key, value): if self.writeback: self.cache[key] = value else: f = StringIO() p = Pickler(f, self._protocol) p.dump(value) self.dict[key] = f.getvalue() seems to match the documentation's intent. (First report, sorry for any formatting/style issues!) -- components: Extension Modules messages: 85971 nosy: jherskovic severity: normal status: open title: Shelve module writeback parameter does not act as advertised type: behavior versions: Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5754 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
