[issue34134] multiprocessing memory huge usage

Antoine Pitrou added the comment: This is basically fixed, except that I'll let the Release Manager choose whether 3.6 gets the fix as well. Thanks! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.7

[issue34134] multiprocessing memory huge usage

Antoine Pitrou added the comment: New changeset c2674bf11036af1e06c1be739f0eebcc72dfbf7a by Antoine Pitrou (Miss Islington (bot)) in branch '3.7': bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) (gh-11673)

[issue34134] multiprocessing memory huge usage

Change by miss-islington : -- pull_requests: +11489, 11490, 11491 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue34134] multiprocessing memory huge usage

Change by miss-islington : -- pull_requests: +11489, 11490 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34134] multiprocessing memory huge usage

Change by miss-islington : -- pull_requests: +11489 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34134] multiprocessing memory huge usage

Change by miss-islington : -- pull_requests: +11487, 11488 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34134] multiprocessing memory huge usage

Change by miss-islington : -- pull_requests: +11487 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34134] multiprocessing memory huge usage

Antoine Pitrou added the comment: New changeset 3bab40db96efda2e127ef84e6501fda0cdc4f5b8 by Antoine Pitrou (Windson yang) in branch 'master': bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324)

[issue34134] multiprocessing memory huge usage

Change by Terry J. Reedy : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34134] multiprocessing memory huge usage

Change by Windson Yang : -- keywords: +patch pull_requests: +7859 stage: -> patch review ___ Python tracker ___ ___

[issue34134] multiprocessing memory huge usage

Xiang Zhang added the comment: I'm +1 for INADA's change, but not more examples trying to distinguish every detail difference. -- ___ Python tracker ___

[issue34134] multiprocessing memory huge usage

Windson Yang added the comment: Yes, we should not. But we can do this when use map function. the document gives a good example but doesn't say much about real differences between map and imap. Maybe we should add some notes like INADA suggest. map function will convert iterable to list if

[issue34134] multiprocessing memory huge usage

Xiang Zhang added the comment: Why accessing the result outside of with block? The pool is terminated while exiting the block before the work is done. -- ___ Python tracker

[issue34134] multiprocessing memory huge usage

Windson Yang added the comment: The code didn't work with imap because imap create a generator, so we can't access result outside the with statement. with Pool(os.cpu_count()) as p: result = p.imap(clean_up, k, 50) for r in result: print(r) In

[issue34134] multiprocessing memory huge usage

Windson Yang added the comment: Thank you Xiang Zhang, I found the code keeps hanging when I use imap, I will try to figure out tomorrow. -- ___ Python tracker ___

[issue34134] multiprocessing memory huge usage

Xiang Zhang added the comment: One thing worth a try here maybe turn `len` to `operator.length_hint`. But I am not sure it's a good idea and just a mention here. -- nosy: +xiang.zhang versions: -Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7

[issue34134] multiprocessing memory huge usage

Windson Yang added the comment: Thank you, I will try to make a pull request and let other to edit it. -- ___ Python tracker ___

[issue34134] multiprocessing memory huge usage

INADA Naoki added the comment: > I think we should add something like "if you are using generator, consider > use imap instead" I think it's not good hint. There are short generator. And there are long (or infinite) iterator other than generator too. Maybe, "if iterator is not sequence

[issue34134] multiprocessing memory huge usage

Windson Yang added the comment: Thank you for the hint, INADA. I think we should add something like "if you are using generator, consider use imap instead" in https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.pool.Pool.map --

[issue34134] multiprocessing memory huge usage

INADA Naoki added the comment: Do you imap or imap_unorderd? They are intended for use with iterator, including generator. -- nosy: +inada.naoki ___ Python tracker ___

[issue34134] multiprocessing memory huge usage

New submission from Windson Yang : I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file) I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383) if not hasattr(iterable, '__len__'):