[issue28699] Imap from ThreadPool behaves unexpectedly

Changes by Xiang Zhang : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: New changeset 9f8e0904580cae3b99f8d343569b76e1be7e6092 by Xiang Zhang in branch '3.5': bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-884) https://github.com/python/cpython/commit/9f8e0904580cae3b99f8d343569b76e1be7e6092 --

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: New changeset 346dcd65e6b832a35b4cfc15b7309b51a38e9ca2 by Xiang Zhang in branch '3.6': bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-882) https://github.com/python/cpython/commit/346dcd65e6b832a35b4cfc15b7309b51a38e9ca2 --

[issue28699] Imap from ThreadPool behaves unexpectedly

Changes by Xiang Zhang : -- pull_requests: +786 ___ Python tracker ___ ___ Python-bugs-list

[issue28699] Imap from ThreadPool behaves unexpectedly

Changes by Xiang Zhang : -- pull_requests: +784 ___ Python tracker ___ ___ Python-bugs-list

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: New changeset 794623bdb232eafd8925f76470209afcdcbcdcd2 by Xiang Zhang in branch 'master': bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-693) https://github.com/python/cpython/commit/794623bdb232eafd8925f76470209afcdcbcdcd2 --

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: Davin, I propose a PR to solve this problem based on your patch. Hope you are willing to review and let's finish this. ;-) -- stage: needs patch -> patch review ___ Python tracker

[issue28699] Imap from ThreadPool behaves unexpectedly

Changes by Xiang Zhang : -- pull_requests: +566 ___ Python tracker ___ ___ Python-bugs-list

[issue28699] Imap from ThreadPool behaves unexpectedly

fiete added the comment: Since the only thing I know about the multiprocessing internals is what I just read in the source code trying to debug my imap_unordered call, I'll add the following example, not knowing whether this is already covered by everything you have until now. import

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: > 4. Guard against misbehaving generators/iterables *before* they are put into > the taskqueue. This approach is good. 2 points about the patch: 1. How about _map_async(map)? Does it need the same strategy? For an iterator with __len__ defined it seems to get

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: Attaching promised crude tests. -- Added file: http://bugs.python.org/file45646/issue_28699_repro.py ___ Python tracker

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: @xiang.zhang: Nice catch -- thank you for pointing out the additional issue that arises when trying to provoke the issue twice in a row. The module attribute `job_counter` helped, in this situation, point out what might have otherwise been overlooked. I didn't

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: > Your patch looks to be introducing a number of changes to the structure of > the data being passed around between threads and even monitored/indirectly > shared across processes. That's why I say even myself don't like it. To solve an edge case some long

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: @xiang.zhang: Your patch looks to be introducing a number of changes to the structure of the data being passed around between threads and even monitored/indirectly shared across processes. It's looking increasingly high risk to me. We already have logic for

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: Here is a patch which is just a try. I don't quite like the implementation but I can not figure out a better solution. The examples in this one and #28696 seems to work and no test fails currently. -- Added file:

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: What's more, this case seems non-reentrant. Since there is no task in this case, the job id is always 0 which is not true. This means after the first time, we can not set even the exception. -- ___ Python tracker

[issue28699] Imap from ThreadPool behaves unexpectedly

Xiang Zhang added the comment: Hi Davin, could it be fixed like this? diff -r 05a728e1da15 Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py Wed Nov 16 16:35:53 2016 -0800 +++ b/Lib/multiprocessing/pool.py Thu Nov 17 16:35:38 2016 +0800 @@ -398,7 +398,7 @@

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: Though it still lacks a proper test, I'm attaching a preliminary patch to address the problematic behavior in 3.5/3.6/default in the hopes that others might help test it. -- keywords: +patch Added file:

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: This inconsistent behavior in imap on both Pool and ThreadPool is not what I would expect. -- versions: +Python 3.6, Python 3.7 ___ Python tracker

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: To tie in the example given by @elias in issue28625, this inconsistency in behavior is not limited to ThreadPool -- it appears with a process Pool as well: from multiprocessing import Pool def double(x): return 2 * x def get_numbers(): raise

[issue28699] Imap from ThreadPool behaves unexpectedly

Changes by Lev Veshnyakov : -- nosy: +davin ___ Python tracker ___ ___

[issue28699] Imap from ThreadPool behaves unexpectedly

New submission from Lev Veshnyakov: Consider the following code: from multiprocessing.pool import ThreadPool pool = ThreadPool(10) def gen(): yield 1 + '1' # here is an error print(list(pool.imap(str, gen( # prints [] print(list(pool.map(str, gen( # raises TypeError The