[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fd32a0e2ee445dd7156323d216627112e66b0a69 by Victor Stinner in 
branch '3.7':
[3.7] bpo-38546: Backport multiprocessing tests fixes from master (GH-19689)
https://github.com/python/cpython/commit/fd32a0e2ee445dd7156323d216627112e66b0a69


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +19009
pull_request: https://github.com/python/cpython/pull/19689

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[miss-islington]

miss-islington  added the comment:


New changeset b8bbdf049b0472b8edc4298bfa61e62e3a584e98 by Miss Islington (bot) 
in branch '3.7':
bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652) 
(GH-17655)
https://github.com/python/cpython/commit/b8bbdf049b0472b8edc4298bfa61e62e3a584e98


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b0eb046cbd0dbb7b17f16aad6de20fac5305f387 by Victor Stinner in 
branch '3.8':
bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652) 
(GH-17655)
https://github.com/python/cpython/commit/b0eb046cbd0dbb7b17f16aad6de20fac5305f387


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[miss-islington]

Change by miss-islington :


--
pull_requests: +17123
pull_request: https://github.com/python/cpython/pull/17656

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

> bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652)

I tested manually on the RHEL8 worker and I confirm that this change prevents 
the reap_children() warning.

I close the issue. The change will be shortly backported to 3.7 and 3.8.

--

I'm not sure if PR 17640 is useful, let's discuss it on the PR directly.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +17122
pull_request: https://github.com/python/cpython/pull/17655

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[miss-islington]

Change by miss-islington :


--
pull_requests: +17121
pull_request: https://github.com/python/cpython/pull/17654

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 673c39331f844a80c465efd7cff88ac55c432bfb by Victor Stinner in 
branch 'master':
bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652)
https://github.com/python/cpython/commit/673c39331f844a80c465efd7cff88ac55c432bfb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[miss-islington]

Change by miss-islington :


--
pull_requests: +17120
pull_request: https://github.com/python/cpython/pull/17653

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +17119
pull_request: https://github.com/python/cpython/pull/17652

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 35acb3597208e10a101140474adec86859d57f61 by Victor Stinner in 
branch '3.8':
bpo-38546: multiprocessing tests stop the resource tracker (GH-17641) (GH-17647)
https://github.com/python/cpython/commit/35acb3597208e10a101140474adec86859d57f61


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +17115
pull_request: https://github.com/python/cpython/pull/17647

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9707e8e22d80ca97bf7a9812816701cecde6d226 by Victor Stinner in 
branch 'master':
bpo-38546: multiprocessing tests stop the resource tracker (GH-17641)
https://github.com/python/cpython/commit/9707e8e22d80ca97bf7a9812816701cecde6d226


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

I can reproduce the issue with the following match file:

test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest.test_no_stale_references
test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest.test_ressources_gced_in_workers
test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest.test_shutdown_race_issue12456

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

Example of failure, with additional debug prints:



0:01:02 load avg: 42.13 [ 38/1] test_concurrent_futures failed (env changed)

  
test_no_stale_references 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
2.00s reap_children 
  
ok  

  
test_ressources_gced_in_workers 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
3.86s reap_children 
   
ok  

  
test_shutdown_race_issue12456 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
2.22s reap_children 
 
ok  

  
test_submit 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
2.26s reap_children 
   
ok  

  
test_submit_keyword 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
1.59s reap_children 
   
ok  

  
test_traceback 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
1.60s reap_children 

ok  

 
MP cleanup: cleanup process 
  
MP cleanup: stop forkserver 
  
MP cleanup: stop res track  

MP cleanup: run finalizers  

reap_children   
  

  
--  
  


Ran 6 tests in 17.210s  


  
OK  
  
reap_children   
  
Warning -- reap_children() reaped child process 4992
 

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

I managed to reproduce the issue on the RHEL8 worker using:

* terminal 1: ./python -u -m test --fail-env-changed test_concurrent_futures -v 
-m ProcessPoolSpawnProcessPoolExecutorTest -F -j 15
* terminal 2: ./python -m test -j4 -u all -F -r

I can still reproduce the issue with PR 17641 fix, so it's not enough.

I also applied PR 17640 and 2 extra changes:

diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py
index 1f9c2daa25..dbbf84b3d2 100644
--- a/Lib/multiprocessing/managers.py
+++ b/Lib/multiprocessing/managers.py
@@ -661,7 +661,7 @@ class BaseManager(object):
 except Exception:
 pass
 
-process.join(timeout=1.0)
+process.join(timeout=0.1)
 if process.is_alive():
 util.info('manager still alive')
 if hasattr(process, 'terminate'):
diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index 215bab8131..110c7f945e 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -2399,6 +2399,7 @@ def reap_children():
 
 # Reap all our dead child processes so we don't leave zombies around.
 # These hog resources and might be causing some of the buildbots to die.
+time.sleep(0.5)
 while True:
 try:
 # Read the exit status of any child process which already completed

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +17109
pull_request: https://github.com/python/cpython/pull/17641

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +17108
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17640

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

New failure on AMD64 RHEL7 LTO 3.x:
https://buildbot.python.org/all/#/builders/103/builds/17

test_interpreter_shutdown
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest)
... 1.61s Warning -- reap_children() reaped child process 9934

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Ran 168 tests in 160.415s
OK (skipped=3)
Warning -- multiprocessing.process._dangling was modified by 
test_concurrent_futures
  Before: set()
  After:  {}

Another failure in AMD64 RHEL8 LTO + PGO 3.x

https://buildbot.python.org/all/#/builders/284/builds/300/steps/5/logs/stdio

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

AMD64 Debian PGO 3.x:
https://buildbot.python.org/all/#/builders/47/builds/3851

test_submit_keyword 
(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ... 
0.62s Warning -- reap_children() reaped child process 32627
Warning -- multiprocessing.process._dangling was modified by 
test_concurrent_futures

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38546] test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

[STINNER Victor]

STINNER Victor  added the comment:

I closed bpo-38448 as duplicate of this issue. Copy of its only message:

AMD64 RHEL8 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/272/builds/23

0:27:13 load avg: 4.88 [416/419/1] test_concurrent_futures failed (env changed) 
(17 min 11 sec) -- running: test_capi (7 min 28 sec), test_gdb (8 min 49 sec), 
test_asyncio (23 min 23 sec)
beginning 6 repetitions
123456
.Warning -- reap_children() reaped child process 26487
.
Warning -- multiprocessing.process._dangling was modified by 
test_concurrent_futures
  Before: set()
  After:  {}

--
title: test_concurrent_futures: test_interpreter_shutdown() warning 
"reap_children() reaped child process" on AMD64 RHEL7 3.x -> 
test_concurrent_futures: reap_children() warnings on RHEL7 and RHEL8 buildbots

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com