[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset 957d0e9b59bd27ca7c473560634d8b5dbe66338c by Victor Stinner in branch 'master': bpo-26762: _test_multiprocessing reports dangling (#3064) https://github.com/python/cpython/commit/957d0e9b59bd27ca7c473560634d8b5dbe66338c --

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +3101 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: Ok, now test_multiprocessing_spawn should not "leak" processes nor threads anymore. I fixed Python 3.6 and master. I close the issue. I don't think that it's worth it to fix tests in 2.7, and 3.5 is entering security-only mode. Note: I will backport

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset d0adfb25c5082046133a18fd185375508c1c334f by Victor Stinner in branch '3.6': [3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 (#2879) https://github.com/python/cpython/commit/d0adfb25c5082046133a18fd185375508c1c334f

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +2930 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: Ok, I pushed 3 changes. Only one test still has a minor issue: test.test_multiprocessing_spawn.WithProcessesTestPool.test_context The bpo-31019 fixes it. Later, we will be able to modify ManagerMixin.tearDownClass() to emit a warning immediately if a manager

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset b4c52966c810b5c5e088fceff403247f610b7d13 by Victor Stinner in branch 'master': bpo-26762: test_multiprocessing close more queues (#2855) https://github.com/python/cpython/commit/b4c52966c810b5c5e088fceff403247f610b7d13 --

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset ffb49408f0780ae80a553208aa133bc5bb3ba129 by Victor Stinner in branch 'master': test_multiprocessing detects dangling per test case (#2841) https://github.com/python/cpython/commit/ffb49408f0780ae80a553208aa133bc5bb3ba129 --

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +2906 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: > test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback Oh, this test only "leaks" dangling processes and threads because PR 2841 was too strict. The problem is that multiprocessing.Pool has thread attributes which hold references to the pool:

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset d7e64d9934d86aa6173229de5af5fe908662a33a by Victor Stinner in branch 'master': test_multiprocessing: Fix dangling process/thread (#2850) https://github.com/python/cpython/commit/d7e64d9934d86aa6173229de5af5fe908662a33a --

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: Using PR 2841, I wrote a tool to run each test in subprocess to check if a test method leaks. I found these methods: ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: PR 2850 + PR 2849 (of bpo-31019) fix most dangling process/thread, but one specific test still leaks: test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback I will try to fix it once other bugs are fixed. --

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: I rewrote PR 2841 to detect more bugs. -- ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +2899 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: I just created bpo-31019 "multiprocessing.Pool should join "dead" processes" to fix dangling processes related to pool, like test.test_multiprocessing_spawn.WithProcessesTestPool.test_context. -- ___ Python tracker

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: Antoine Pitrou: "I'm not able to reproduce. Can you try again with latest master?" Oh, maybe I found why... There is a tearDownModule() function which hides bugs. Try following change: diff --git a/Lib/test/_test_multiprocessing.py

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: If you want to use --fail-env-changed with PR 2841, for example to use test.bisect, you may want this extra patch: diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index f2342c3..5f9e5c3 100644 ---

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: New changeset 06634950c553f8df83330ed468c11483b857b7dc by Victor Stinner in branch 'master': bpo-26762: Avoid daemon process in _test_multiprocessing (#2842) https://github.com/python/cpython/commit/06634950c553f8df83330ed468c11483b857b7dc --

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: With PR 2841 + PR 2842, there is still at least one test which leaks processes: --- $ ./python -m test --match=test_notify_all -v test_multiprocessing_spawn == CPython 3.7.0a0 (heads/master:b364d9f, Jul 24 2017, 11:06:33) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: > With this PR, the following test fails: Ah, it seems like test_level() alone is enough to reproduce the bug and the following PR fixes it: https://github.com/python/cpython/pull/2842 -- ___ Python tracker

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +2886 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: I'm not sure that the issue is really fixed. I applied my msg263461 patch and I still see the list of "python" processes growing but then also a little bit decreasing, and some zombi processes. The problem is that it's hard to guess if a growing "ps" list is

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by STINNER Victor : -- pull_requests: +2885 ___ Python tracker ___ ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

Antoine Pitrou added the comment: I'm not able to reproduce. Can you try again with latest master? -- nosy: +pitrou ___ Python tracker ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

STINNER Victor added the comment: Using this patch, I can see the list of "python" processes slowly growing: diff -r ad5b079565ad Lib/unittest/case.py --- a/Lib/unittest/case.py Tue Apr 12 23:15:44 2016 -0700 +++ b/Lib/unittest/case.py Fri Apr 15 11:06:07 2016 +0200 @@ -10,6 +10,7 @@

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by Serhiy Storchaka : -- nosy: +haypo, serhiy.storchaka ___ Python tracker ___

[issue26762] test_multiprocessing_spawn leaves processes running in background

Changes by Ned Deily : -- nosy: +davin ___ Python tracker ___ ___ Python-bugs-list mailing

[issue26762] test_multiprocessing_spawn leaves processes running in background

New submission from Martin Panter: I noticed that this test leaves processes running in the background for a moment after the parent Python process exits. They disappear pretty quickly, but even so, it seems like a bad design. However I am not familiar with the multiprocessing module, so