[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset 957d0e9b59bd27ca7c473560634d8b5dbe66338c by Victor Stinner in 
branch 'master':
bpo-26762: _test_multiprocessing reports dangling (#3064)
https://github.com/python/cpython/commit/957d0e9b59bd27ca7c473560634d8b5dbe66338c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3101

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

Ok, now test_multiprocessing_spawn should not "leak" processes nor threads 
anymore. I fixed Python 3.6 and master. I close the issue.

I don't think that it's worth it to fix tests in 2.7, and 3.5 is entering 
security-only mode.

Note: I will backport bpo-31019 to Python 2.7, since this issue changes the 
code, not tests.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset d0adfb25c5082046133a18fd185375508c1c334f by Victor Stinner in 
branch '3.6':
[3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 
(#2879)
https://github.com/python/cpython/commit/d0adfb25c5082046133a18fd185375508c1c334f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2930

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

Ok, I pushed 3 changes. Only one test still has a minor issue:

test.test_multiprocessing_spawn.WithProcessesTestPool.test_context

The bpo-31019 fixes it.

Later, we will be able to modify ManagerMixin.tearDownClass() to emit a warning 
immediately if a manager leaks a process, and not have this kind loop to sleep 
until all processes completed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset b4c52966c810b5c5e088fceff403247f610b7d13 by Victor Stinner in 
branch 'master':
bpo-26762: test_multiprocessing close more queues (#2855)
https://github.com/python/cpython/commit/b4c52966c810b5c5e088fceff403247f610b7d13


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset ffb49408f0780ae80a553208aa133bc5bb3ba129 by Victor Stinner in 
branch 'master':
test_multiprocessing detects dangling per test case (#2841)
https://github.com/python/cpython/commit/ffb49408f0780ae80a553208aa133bc5bb3ba129


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2906

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

> test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback

Oh, this test only "leaks" dangling processes and threads because PR 2841 was 
too strict. The problem is that multiprocessing.Pool has thread attributes 
which hold references to the pool: reference cycle. I don't see any easy way to 
break these cycles; since I don't understand if a poll is still supposed to be 
usable after terminate() or not.

Anyway, my intent here was to do a first cleanup. So I modified my PR 2841 to 
call support.gc_collect() to break reference cycles.

If someone wants to break reference cycles, please open a new issue. IMHO this 
issue already became enough complex :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset d7e64d9934d86aa6173229de5af5fe908662a33a by Victor Stinner in 
branch 'master':
test_multiprocessing: Fix dangling process/thread (#2850)
https://github.com/python/cpython/commit/d7e64d9934d86aa6173229de5af5fe908662a33a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

Using PR 2841, I wrote a tool to run each test in subprocess to check if a test 
method leaks. I found these methods:

 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestBarrier.test_thousand
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestCondition.test_notify
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestEvent.test_event
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestPool.test_context
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestProcess.test_close
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestQueue.test_fork
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestQueue.test_qsize
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestQueue.test_queue_feeder_donot_stop_onexc
 ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match 
test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

PR 2850 + PR 2849 (of bpo-31019) fix most dangling process/thread, but one 
specific test still leaks:

test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback

I will try to fix it once other bugs are fixed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

I rewrote PR 2841 to detect more bugs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2899

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

I just created bpo-31019 "multiprocessing.Pool should join "dead" processes" to 
fix dangling processes related to pool, like 
test.test_multiprocessing_spawn.WithProcessesTestPool.test_context.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

Antoine Pitrou: "I'm not able to reproduce.  Can you try again with latest 
master?"

Oh, maybe I found why... There is a tearDownModule() function which hides bugs. 
Try following change:

diff --git a/Lib/test/_test_multiprocessing.py 
b/Lib/test/_test_multiprocessing.py
index 88e2eb3..ae77468 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -4432,8 +4432,8 @@ def install_tests_in_module_dict(remote_globs, 
start_method):
 def tearDownModule():
 multiprocessing.set_start_method(old_start_method[0], force=True)
 # pause a bit so we don't get warning about dangling threads/processes
-time.sleep(0.5)
-multiprocessing.process._cleanup()
+# time.sleep(0.5)
+# multiprocessing.process._cleanup()
 gc.collect()
 tmp = set(multiprocessing.process._dangling) - set(dangling[0])
 if tmp:

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

If you want to use --fail-env-changed with PR 2841, for example to use 
test.bisect, you may want this extra patch:

diff --git a/Lib/test/_test_multiprocessing.py 
b/Lib/test/_test_multiprocessing.py
index f2342c3..5f9e5c3 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -4352,6 +4352,7 @@ class ManagerMixin(object):
   "children after %s seconds"
   % (multiprocessing.active_children(), dt),
   file=sys.stderr)
+test.support.environment_altered = True
 break
 
 gc.collect()   # do garbage collection
@@ -4359,6 +4360,7 @@ class ManagerMixin(object):
 # This is not really an error since some tests do not
 # ensure that all processes which hold a reference to a
 # managed object have been joined.
+test.support.environment_altered = True
 print('Warning -- Shared objects which still exist at manager '
   'shutdown:')
 print(cls.manager._debug_info())

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:


New changeset 06634950c553f8df83330ed468c11483b857b7dc by Victor Stinner in 
branch 'master':
bpo-26762: Avoid daemon process in _test_multiprocessing (#2842)
https://github.com/python/cpython/commit/06634950c553f8df83330ed468c11483b857b7dc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

With PR 2841 + PR 2842, there is still at least one test which leaks processes:

---
$ ./python -m test --match=test_notify_all -v test_multiprocessing_spawn
== CPython 3.7.0a0 (heads/master:b364d9f, Jul 24 2017, 11:06:33) [GCC 6.3.1 
20161221 (Red Hat 6.3.1-1)]
== Linux-4.11.9-200.fc25.x86_64-x86_64-with-fedora-25-Twenty_Five little-endian
== hash algorithm: siphash24 64bit
== cwd: /home/haypo/prog/python/master/build/test_python_28469
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, 
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, 
verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 load avg: 0.44 [1/1] test_multiprocessing_spawn
test_notify_all (test.test_multiprocessing_spawn.WithManagerTestCondition) ... 
ok
test_notify_all (test.test_multiprocessing_spawn.WithProcessesTestCondition) 
... ok
Warning -- reap_children() reaped child process 28553
Warning -- reap_children() reaped child process 28557
Warning -- reap_children() reaped child process 28559
Warning -- reap_children() reaped child process 28561
test_notify_all (test.test_multiprocessing_spawn.WithThreadsTestCondition) ... 
ok
Dangling processes: {, 
, , 
}

--
Ran 3 tests in 3.772s

OK
Warning -- multiprocessing.process._dangling was modified by 
test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0x7f2385200400>
  After:  <_weakrefset.WeakSet object at 0x7f238a8ff260> 
test_multiprocessing_spawn failed (env changed)

1 test altered the execution environment:
test_multiprocessing_spawn

Total duration: 4 sec
Tests result: SUCCESS
---

I looked at the code of the test: it's full of "p.daemon = True" and "t.daemon 
= True", whereas these processes and threads are not tracked in a list to be 
cleared manually, so I'm not surprised of the warnings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

> With this PR, the following test fails:

Ah, it seems like test_level() alone is enough to reproduce the bug and the 
following PR fixes it:
https://github.com/python/cpython/pull/2842

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2886

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

I'm not sure that the issue is really fixed. I applied my msg263461 patch and I 
still see the list of "python" processes growing but then also a little bit 
decreasing, and some zombi  processes.

The problem is that it's hard to guess if a growing "ps" list is a bug or not, 
since we only cleanup things in tearDownClass(), not in tearDown(). So it's ok 
to keep some child processes between two unit tests.

I wrote a first change to help debugging the issue. My change adds more 
warnings and calls support.reap_children():
https://github.com/python/cpython/pull/2841

With this PR, the following test fails:
---
haypo@selma$ ./python -u -m test -v --fail-env-changed 
--match=test.test_multiprocessing_spawn.WithThreadsTestPool.test_release_task_refs
 --match=test.test_multiprocessing_spawn.WithProcessesTestLogging.test_level 
test_multiprocessing_spawn 
== CPython 3.7.0a0 (heads/master:b364d9f, Jul 24 2017, 11:06:33) [GCC 6.3.1 
20161221 (Red Hat 6.3.1-1)]
== Linux-4.11.9-200.fc25.x86_64-x86_64-with-fedora-25-Twenty_Five little-endian
== hash algorithm: siphash24 64bit
== cwd: /home/haypo/prog/python/master/build/test_python_24433
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, 
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, 
verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 load avg: 0.27 [1/1] test_multiprocessing_spawn
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
Warning -- reap_children() reaped child process 24437
test_release_task_refs (test.test_multiprocessing_spawn.WithThreadsTestPool) 
... ok
Warning -- reap_children() reaped child process 24438
Dangling processes: {, }

--
Ran 2 tests in 1.369s

OK
Warning -- multiprocessing.process._dangling was modified by 
test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0x7fdc24f70058>
  After:  <_weakrefset.WeakSet object at 0x7fdc2a666260> 
test_multiprocessing_spawn failed (env changed)

1 test altered the execution environment:
test_multiprocessing_spawn

Total duration: 2 sec
Tests result: ENV CHANGED
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2885

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[Antoine Pitrou]

Antoine Pitrou added the comment:

I'm not able to reproduce.  Can you try again with latest master?

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[STINNER Victor]

STINNER Victor added the comment:

Using this patch, I can see the list of "python" processes slowly growing:

diff -r ad5b079565ad Lib/unittest/case.py
--- a/Lib/unittest/case.py  Tue Apr 12 23:15:44 2016 -0700
+++ b/Lib/unittest/case.py  Fri Apr 15 11:06:07 2016 +0200
@@ -10,6 +10,7 @@ import warnings
 import collections
 import contextlib
 import traceback
+import os
 
 from . import result
 from .util import (strclass, safe_repr, _count_diff_all_purpose,
@@ -590,6 +591,7 @@ class TestCase(object):
 expecting_failure = expecting_failure_class or expecting_failure_method
 outcome = _Outcome(result)
 try:
+os.system("ps")
 self._outcome = outcome
 
 with outcome.testPartExecutor(self):
@@ -614,6 +616,7 @@ class TestCase(object):
 self._addUnexpectedSuccess(result)
 else:
 result.addSuccess(self)
+os.system("ps")
 return result
 finally:
 result.stopTest(self)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +haypo, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[Ned Deily]

Changes by Ned Deily :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26762] test_multiprocessing_spawn leaves processes running in background

[Martin Panter]

New submission from Martin Panter:

I noticed that this test leaves processes running in the background for a 
moment after the parent Python process exits. They disappear pretty quickly, 
but even so, it seems like a bad design. However I am not familiar with the 
multiprocessing module, so maybe this is unavoidable.

$ ps
  PID TTY  TIME CMD
  597 pts/200:00:01 bash
13423 pts/200:00:00 ps
$ python3.5 -m test test_multiprocessing_spawn; ps
[1/1] test_multiprocessing_spawn
1 test OK.
  PID TTY  TIME CMD
  597 pts/200:00:01 bash
13429 pts/200:00:00 python3.5
13475 pts/200:00:00 python3.5
15066 pts/200:00:00 ps

--
components: Tests
messages: 263442
nosy: martin.panter
priority: normal
severity: normal
status: open
title: test_multiprocessing_spawn leaves processes running in background
type: behavior
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com