[issue27144] concurrent.futures.as_completed() memory inefficiency

STINNER Victor added the comment: Thank you both for this nice enhancement. -- ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: The regression should be fixed now. -- resolution: -> fixed status: open -> closed ___ Python tracker ___ _

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: New changeset 5cbca0235b8da07c9454bcaa94f12d59c2df0ad2 by Antoine Pitrou in branch '3.6': [3.6] Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (GH-3270) (#3271) https://github.com/python/cpython/commit/5cbca0235b8da07c9454bcaa94f12d59c2df0

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Antoine Pitrou : -- pull_requests: +3315 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: New changeset 2ef37607b7aacb7c750d008b9113fe11f96163c0 by Antoine Pitrou in branch 'master': Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (#3270) https://github.com/python/cpython/commit/2ef37607b7aacb7c750d008b9113fe11f96163c0

[issue27144] concurrent.futures.as_completed() memory inefficiency

Grzegorz Grzywacz added the comment: Tests are reusing finished futures. `_yield_and_decref` function do not clear waiters in finished futures. In the initial merge i propose to clear waiters but after review we decide it should be removed. I am confused now, should we change tests or restore

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Antoine Pitrou : -- pull_requests: +3314 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: Thank you Victor. There is indeed a logic error in the new code. I'm opening a new PR. -- ___ Python tracker ___

[issue27144] concurrent.futures.as_completed() memory inefficiency

STINNER Victor added the comment: "concurrent.futures.as_completed() memory inefficiency" hum, sadly the commit 97e1b1c81458d2109b2ffed32ffa1eb643a6c3b9 introduced a reference leak. Example: $ ./python -m test -R 3:3 -test_concurrent_futures -m test.test_concurrent_futures.ProcessPoolAsComplet

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: I've merged the PR to master and backported it to 3.6. Thank you Grzegorz for contributing this! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: -Python 3.5 ___ Python trac

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: New changeset ea767915f7476c1fe97f7b1a53304d57f105bdd2 by Antoine Pitrou in branch '3.6': [3.6] bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (GH-1560) (#3266) https://github.com/python/cpython/commit/ea76

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Antoine Pitrou : -- pull_requests: +3310 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue27144] concurrent.futures.as_completed() memory inefficiency

Antoine Pitrou added the comment: New changeset 97e1b1c81458d2109b2ffed32ffa1eb643a6c3b9 by Antoine Pitrou (Grzegorz Grzywacz) in branch 'master': bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (#1560) https://github.com/python/cpython/comm

[issue27144] concurrent.futures.as_completed() memory inefficiency

Grzegorz Grzywacz added the comment: > We just ran into the exact same issue here in Google using a > ThreadPoolExecutor.map call Looks like map got simillar issue. I created GitHub PR with map fixed. > Concurrent package was added in 3.2. How backport it 2.7? There is official, unofficial 2.7

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Grzegorz Grzywacz : -- pull_requests: +1658 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue27144] concurrent.futures.as_completed() memory inefficiency

Xiang Zhang added the comment: > Any update on this reviewing, patching, and then backporting to 2.7? concurrent package was added in 3.2. How backport to 2.7? Nosy myself. -- nosy: +xiang.zhang versions: +Python 3.7 ___ Python tracker

[issue27144] concurrent.futures.as_completed() memory inefficiency

Mark DePristo added the comment: Any update on this reviewing, patching, and then backporting to 2.7? We've just run into the exact same issue here in Google using a ThreadPoolExecutor.map call growing memory usage without bounds due to the Future holding onto its result even after being retur

[issue27144] concurrent.futures.as_completed() memory inefficiency

Will Vousden added the comment: Is there a reason this hasn't been merged yet? Fixing this bug locally (albeit just by setting Future._result = None when I've extracted the result) reduced the memory footprint of my program from 50 GB to 7 GB, so it seems worth it. -- ___

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by willvousden : -- nosy: +willvousden ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Rob Nester : -- nosy: +rnester ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.o

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by SilentGhost : -- stage: -> patch review versions: -Python 3.2, Python 3.3, Python 3.4 ___ Python tracker ___ ___ Python-b

[issue27144] concurrent.futures.as_completed() memory inefficiency

Changes by Grzegorz Grzywacz : -- keywords: +patch Added file: http://bugs.python.org/file43038/issue27144.patch ___ Python tracker ___ __

[issue27144] concurrent.futures.as_completed() memory inefficiency

New submission from Grzegorz Grzywacz: as_complite generator keeps reference of all passed futures until StopIteration. It may lead to serious memory inefficiency. Solution is to remove reference from lists and yield future ad-hoc. I have submitted patch and reproduce sample. I can create bac