[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

New submission from Alon Diamant:

When imap() or imap_unordered() are called with the iterable parameter set as a 
generator function, and when that generator function raises an exception, then 
the _task_handler thread (running the method _handle_tasks) dies immediately, 
without causing the other threads to stop and without reporting the exception 
to the main thread (that one that called imap()).

I saw this issue in Python 2.7.8, 2.7.9 and 3.4.2. Didn't check other versions, 
but I assume this is a bug in all Python versions since 2.6.

I will be attaching examples that reproduce this issue, as well as patches for 
both Python 2.7 and Python 3.4.

--
components: Library (Lib)
messages: 232643
nosy: advance512
priority: normal
severity: normal
status: open
title: multiprocessing.pool methods imap() and imap_unordered() cause deadlock
type: behavior
versions: Python 2.7, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


Added file: http://bugs.python.org/file37449/Issue_23051_reproducer_v2_7.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


--
keywords: +patch
Added file: http://bugs.python.org/file37450/Issue_23051_fix_v2_7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


Added file: http://bugs.python.org/file37451/Issue_23051_reproducer_v3_4.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


Added file: http://bugs.python.org/file37452/Issue_23051_fix_v3_4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


Removed file: http://bugs.python.org/file37452/Issue_23051_fix_v3_4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Changes by Alon Diamant diamant.a...@gmail.com:


Added file: http://bugs.python.org/file37453/Issue_23051_fix_v3_4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23051] multiprocessing.pool methods imap() and imap_unordered() cause deadlock

[Alon Diamant]

Alon Diamant added the comment:

The patches I attached do 2 things:

1. A deadlock is prevented, wherein the main thread waits forever for the Pool 
thread/s to finish their execution, while they wait for instructions to 
terminate from the _task_handler thread which has died. Instead, the exception 
are caught and handled and termination of the pool execution is performed.
2. The exception that was raised is caught and passed to the main thread, and 
is re-thrown in the context of the main thread - hence the user catch it and 
handle it, or - at the very least - be aware of the issue.

I tested the patch to the best of my abilities, and am almost certain nothing 
was changed performance wise nor anything broken. 

Further eyes would, of course, only help for confirming this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23051
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com