[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Brian Curtin br...@python.org: -- nosy: -brian.curtin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Steven Myint c...@stevenmyint.com: -- nosy: +myint ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Markus Unterwaditzer added the comment: Can this issue or #9205 be reopened as this particular instance of the problem doesn't seem to be resolved? I still seem to need the workaround from http://stackoverflow.com/a/1408476 -- nosy: +untitaker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Ned Deily n...@acm.org: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Martin Dengler mar...@martindengler.com: -- nosy: +mdengler ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: Closing, as Andrey Vlasovskikh has agreed that this is a duplicate of #9205. -- nosy: +vinay.sajip resolution: - duplicate status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Antoine Pitrou pit...@free.fr added the comment: Note that #9205 fixed concurrent.futures, but not multiprocessing.Pool which is a different kettle of fish. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Gökçen Eraslan gok...@pardus.org.tr: -- nosy: +Gökçen.Eraslan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Greg Brockman g...@mit.edu: -- nosy: +gdb ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh andrey.vlasovsk...@gmail.com added the comment: On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one. Yes, SystemExit should cancel all the tasks that are currently in the queue. I guess my patch doesn't handle this properly. If we keep that behavior, the real problem here is that the result handler hangs if the process that reserved a job is gone, which is going to be handled by #9205. Should we mark it as a duplicate? Yes, I think that #9205 covers this issue, so #8296 may be marked as a duplicate. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Ask Solem a...@opera.com added the comment: On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one. If we keep that behavior, the real problem here is that the result handler hangs if the process that reserved a job is gone, which is going to be handled by #9205. Should we mark it as a duplicate? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Jesse Noller jnol...@gmail.com added the comment: If we keep that behavior, the real problem here is that the result handler hangs if the process that reserved a job is gone, which is going to be handled by #9205. Should we mark it as a duplicate? I would tend to agree with your assessment; we're better served just gracefully handling everything per 9205 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Albert Strasheim full...@gmail.com: -- nosy: +Albert.Strasheim ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Albert Strasheim full...@gmail.com added the comment: Any chance of getting this patch applied? Thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Brian Curtin cur...@acm.org: -- nosy: +asksol ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Ask Solem a...@opera.com added the comment: This is related to our discussions at #9205 as well (http://bugs.python.org/issue9205), as the final patch there will also fix this issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh andrey.vlasovsk...@gmail.com added the comment: Despite of several workarounds available on the Web, the problem persists. Almost any exception that is rised in `worker` function while putting or getting tasks from queues result in Pool hang up. Currently, `worker` is only aware of Exception descendants rised inside of the map function parameter. I've written a unit test that checks if KeyboardInterrupts are handled normally. The source code may be included in `Lib/test/test_multiprocessing.py`. -- Added file: http://bugs.python.org/file16784/test_map_keyboard_interrput.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Andrey Vlasovskikh andrey.vlasovsk...@gmail.com: Removed file: http://bugs.python.org/file16743/test_pool_keyboardinterrupt.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh andrey.vlasovsk...@gmail.com added the comment: Here is a patch that fixes this problem. Basically, it catches all the BaseExceptions that could happen during: a) getting a task from the `inqueue`, b) calling a user function, c) putting a task into the `outqueue`. The exception handler puts the exception to the `outqueue`. It can be cleanly applied on top of revision 78790. -- keywords: +patch Added file: http://bugs.python.org/file16785/fix-sigint.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Brian Curtin cur...@acm.org: -- nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
New submission from Andrey Vlasovskikh andrey.vlasovsk...@gmail.com: multiprocessing.Pool methods map, imap, etc. are said to be able to normally handle exceptions. But it seems that it is true only for synchronous exceptions inside their first func arguments. When (typically during a long-running parallel map) a user hits ^C, an asynchronous KeyboardInterrupt isn't handled properly and leads to the interpreter hangup. More precisely, children processes become defunct (on Linux), so the only way to terminate the whole program is to issue the KILL signal. As stopping a program with ^C while running potentially long parallel computations is probably quite a common scenario, the interpreter should not hang up in such a case. I'm using Python 2.6.5 (r265:79063, Mar 23 2010, 04:44:21) [GCC 4.4.3] on linux2. I've also tried to use the current multiprocessing.pool module from the current (2.7) trunk with my 2.6.5 installation, but the bug persists. -- components: Library (Lib) messages: 102219 nosy: vlasovskikh severity: normal status: open title: multiprocessing.Pool hangs when issuing KeyboardInterrupt type: crash versions: Python 2.6, Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Brian Curtin cur...@acm.org added the comment: Do you have a test case which can reproduce the issue? -- nosy: +brian.curtin stage: - test needed type: crash - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh andrey.vlasovsk...@gmail.com added the comment: Yes, here is my test case. -- Added file: http://bugs.python.org/file16743/test_pool_keyboardinterrupt.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Brian Curtin cur...@acm.org added the comment: You might want to take a look here: http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh andrey.vlasovsk...@gmail.com added the comment: Yes, I've come up with the same solution by myself, but it cannot cover all the cases of the bug. It works only for cases when ^C is hit during a call to the users' function: http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool/2561809#2561809 If the user is lucky, he may hit ^C during getting or putting data into the queues in multiprocessing.pool.worker. To reproduce such a case, you may insert `sleep(10)` before `task = get()` or `put((job, i, result))`, for example. I've encountered such cases just by running test examples several times. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
