[issue22853] Multiprocessing.Queue._feed deadlocks on import
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - serhiy.storchaka nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Serhiy Storchaka added the comment: Oh, right. Harmless. Thanks for all the fish Mark. I meant the bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Serhiy Storchaka added the comment: I'm not sure there is a need to fix this issue. Using multiprocessing Queue at import time looks as yet one way to shoot yourself in the foot. But the patch looks harmful, I'll commit it. -- components: +Library (Lib) stage: patch review - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Mark Lawrence added the comment: That's all we need, harmful patches being committed :) Harmless possibly? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Roundup Robot added the comment: New changeset 069c13ca7a70 by Serhiy Storchaka in branch '2.7': Issue #22853: Fixed a deadlock when use multiprocessing.Queue at import time. https://hg.python.org/cpython/rev/069c13ca7a70 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Davin Potts added the comment: Corrected patch for 3.4 and default/3.5 -- newly introduced test is now turned on this time and the dangling weak references are properly addressed as well as the reference to Empty. Nastiness. Good save, Serhiy. -- Added file: http://bugs.python.org/file38361/issue_22853_revised_only_test_import_lock_in_queue_py34_and_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Roundup Robot added the comment: New changeset cf12856bde17 by Serhiy Storchaka in branch '3.4': Issue #22853: Added regression test for using multiprocessing.Queue at import https://hg.python.org/cpython/rev/cf12856bde17 New changeset dcd6d41f2c9a by Serhiy Storchaka in branch 'default': Issue #22853: Added regression test for using multiprocessing.Queue at import https://hg.python.org/cpython/rev/dcd6d41f2c9a -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Serhiy Storchaka added the comment: Thank you Florian and Davin for your contribution. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Serhiy Storchaka added the comment: As for 3.x, underscored test does not run, and when remove the underscore it runs, but produce a warning (you should run regrtests with -vv to see detailed warnings): $ ./python -m test.regrtest -vv -m '*no_import_lock_contention*' test_multiprocessing_spawn ... Warning -- threading._dangling was modified by test_multiprocessing_spawn Before: _weakrefset.WeakSet object at 0xb6a960ac After: _weakrefset.WeakSet object at 0xb6c4cc0c 1 test altered the execution environment: test_multiprocessing_spawn -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Davin Potts added the comment: Attaching a patch for 2.7 that applies Florian's fix and provides a test for it as well. Although the issue is not triggered on 3.4 or default (3.5), there is the potential for regression there -- attaching a single patch that works for both 3.4 and 3.5 to provide a regression test (only a test, nothing to fix). These patches have been tested on OS X 10.10 and Ubuntu 12.04.5 64-bit for each of 2.7, 3.4, and default (3.5). -- keywords: +patch stage: - patch review Added file: http://bugs.python.org/file38272/issue_22853_fix_and_test_import_lock_in_queue_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Changes by Davin Potts pyt...@discontinuity.net: Added file: http://bugs.python.org/file38273/issue_22853_only_test_import_lock_in_queue_py34_and_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Mark Lawrence added the comment: @Davin I believe that you're interested in multiprocessing issues. -- nosy: +BreamoreBoy, davin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Davin Potts added the comment: Confirmed that the issue can be reproduced under 2.7.9 on OS X 10.10. It is not possible to reproduce the issue with default (3.5) -- taking a look at what's different there, notably the import of is_exiting has moved to the top of the queues module and is no longer only imported at the time Queue._feed is invoked, which is just as Florian advocates doing for 2.7. I should take a closer look to understand what else has changed. @Mark: Cool -- thanks. -- type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
New submission from Florian Finkernagel: If you import a module that creates a multiprocessing.Queue, puts a value, and then waits for to be received again from the queue, you run into a deadlock. The issue is that Queue._feed does 'from .util import is_existing' - which needs the import lock, but is still being held by the main thread. Attached a script that illustrates this. Patch is a two line change, import is_exiting in line 49, remove the import inside the thread: 49c49 from multiprocessing.util import debug, info, Finalize, register_after_fork --- from multiprocessing.util import debug, info, Finalize, register_after_fork, is_exiting 232d231 from .util import is_exiting -- files: show_queue_import_bug.py messages: 231073 nosy: ffinkernagel priority: normal severity: normal status: open title: Multiprocessing.Queue._feed deadlocks on import versions: Python 2.7 Added file: http://bugs.python.org/file37185/show_queue_import_bug.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22853] Multiprocessing.Queue._feed deadlocks on import
Changes by Ned Deily n...@acm.org: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22853 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
