[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset b894b669c98cc365b84cbb8d20f531f1d0686f59 by Victor Stinner in branch '3.7': Update libregrtest from master (GH-19517) https://github.com/python/cpython/commit/b894b669c98cc365b84cbb8d20f531f1d0686f59 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 by Victor Stinner in branch '3.8': [3.8] Update libregrtest from master (GH-19516) https://github.com/python/cpython/commit/67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset a661392f8fb5ac4fc095aa1845d1eb7a25c4e9be by Victor Stinner in branch 'master': bpo-37531: regrtest now catchs ProcessLookupError (GH-16827) https://github.com/python/cpython/commit/a661392f8fb5ac4fc095aa1845d1eb7a25c4e9be --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +16377 pull_request: https://github.com/python/cpython/pull/16827 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: The initial issue has been fixed in branches 3.7, 3.8 and master. I close the issue. Thanks to everybody who help to make these changes possible and helped to debug regressions. It's not perfect, regrtest has to workaround bpo-38207 bug, but at least

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

miss-islington added the comment: New changeset 359a1975cbca488ccd5ea13bd7268d7e88664078 by Miss Islington (bot) in branch '3.8': bpo-37531: regrtest ignores output on timeout (GH-16659) https://github.com/python/cpython/commit/359a1975cbca488ccd5ea13bd7268d7e88664078 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

miss-islington added the comment: New changeset 598bfa4d457d61431142ad99ecbb9bd10cf314e6 by Miss Islington (bot) in branch '3.7': bpo-37531: regrtest ignores output on timeout (GH-16659) https://github.com/python/cpython/commit/598bfa4d457d61431142ad99ecbb9bd10cf314e6 -- nosy:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by miss-islington : -- pull_requests: +16257 pull_request: https://github.com/python/cpython/pull/16676 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by miss-islington : -- pull_requests: +16259 pull_request: https://github.com/python/cpython/pull/16677 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 0ec618af98ac250a91ee9c91f8569e6df6772758 by Victor Stinner in branch 'master': bpo-37531: regrtest ignores output on timeout (GH-16659) https://github.com/python/cpython/commit/0ec618af98ac250a91ee9c91f8569e6df6772758 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +16242 pull_request: https://github.com/python/cpython/pull/16659 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by Kubilay Kocak : -- nosy: +koobs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: > I don't understand why regrtest got a timeout of 60 seconds? Oh, that comes from "timeout = PROGRESS_UPDATE" with "PROGRESS_UPDATE = 30.0". But it means that the main process was stuck for longer than 60 seconds!? The main process is supposed to write an

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Oh, regrtest main process was killed by faulthandler watchdog: it was blocked in _get_result() for longer than 1 minute. Extract of _get_result(): use_faulthandler = (self.ns.timeout is not None) timeout = PROGRESS_UPDATE while True:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 5e1400a6bcbb3350a6665176980a2b8343075c63 by Victor Stinner in branch '3.7': bpo-37531: sync regrtest with master branch (GH-16285) (GH-16289) https://github.com/python/cpython/commit/5e1400a6bcbb3350a6665176980a2b8343075c63 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15874 pull_request: https://github.com/python/cpython/pull/16289 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset fb7746d5d10ec4a34198da672018ba15f5667079 by Victor Stinner in branch '3.8': bpo-37531: sync regrtest with master branch (GH-16285) https://github.com/python/cpython/commit/fb7746d5d10ec4a34198da672018ba15f5667079 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15870 pull_request: https://github.com/python/cpython/pull/16285 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset b2dd2dd6e9212ba6b03885b998736a180ef283d6 by Victor Stinner in branch 'master': bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on all platforms (GH-16282)

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15867 pull_request: https://github.com/python/cpython/pull/16282 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: It seems like the new regrtest design doesn't work as I expected. Calling popen.kill() + popen.stdout.close() + popen.stderr.close() from a thread B does not always interrupt popen.communicate() in thread A. See https://bugs.python.org/issue38207#msg352729

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Oh wait, test_regrtest.test_multiprocessing_timeout() now also hangs on FreeBSD? :-( AMD64 FreeBSD CURRENT Shared 3.8: https://buildbot.python.org/all/#/builders/212/builds/226 running: test_regrtest (24 min 52 sec) 0:32:54 load avg: 0.26 [423/423/1]

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset b9877cd2cc47b6f3512c171814c4f630286279b9 by Victor Stinner in branch 'master': bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on Windows (GH-16247) https://github.com/python/cpython/commit/b9877cd2cc47b6f3512c171814c4f630286279b9

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15843 pull_request: https://github.com/python/cpython/pull/16247 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: I created bpo-38207 "subprocess: On Windows, Popen.kill() + Popen.communicate() is blocking until all processes using the pipe close the pipe". -- ___ Python tracker

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 6591b4bbb1c0b9c26b99e4b2dba1e5cc8546732d by Victor Stinner (Miss Islington (bot)) in branch '3.7': bpo-37531: regrtest main process uses shorter timeout (GH-16220) (GH-16223)

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 5f1590d5e679f4dd0b611ef54ae512f137e78f1b by Victor Stinner (Miss Islington (bot)) in branch '3.8': bpo-37531: regrtest main process uses shorter timeout (GH-16220) (GH-16224)

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by miss-islington : -- pull_requests: +15824 pull_request: https://github.com/python/cpython/pull/16224 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by miss-islington : -- pull_requests: +15823 pull_request: https://github.com/python/cpython/pull/16223 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 46b0b81220a23bc4aee5ba3ba67e8cf1b5df7960 by Victor Stinner in branch 'master': bpo-37531: regrtest main process uses shorter timeout (GH-16220) https://github.com/python/cpython/commit/46b0b81220a23bc4aee5ba3ba67e8cf1b5df7960 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15821 pull_request: https://github.com/python/cpython/pull/16220 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: https://buildbot.python.org/all/#/builders/58/builds/2987 1:06:55 load avg: 3.39 [343/419/1] test_regrtest crashed (Exit code 1) Timeout (0:35:00)! Thread 0x0cdc (most recent call first): File

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: See also bpo-37424. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: Another day, another stuck test_concurrent_futures... https://buildbot.python.org/all/#/builders/40/builds/3030 The test process is again killed (line 568) but the processes from the multiprocess pool are still alive. Once I manually kill those pool

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset d42a4fdc630c54352701a466a9e512bee68b5c48 by Victor Stinner in branch '3.8': bpo-37531: Enhance regrtest multiprocess timeout (GH-15345) (GH-15871) https://github.com/python/cpython/commit/d42a4fdc630c54352701a466a9e512bee68b5c48 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 494b61aeec8e583755c8ff3a147a58657353bbc3 by Victor Stinner in branch '3.7': [3.7] bpo-37531: Enhance regrtest multiprocess timeout (GH-15345) (GH-15874) https://github.com/python/cpython/commit/494b61aeec8e583755c8ff3a147a58657353bbc3

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15514 pull_request: https://github.com/python/cpython/pull/15874 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15513 pull_request: https://github.com/python/cpython/pull/15871 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: Well, the kill timeout doesn't seem to be working, at least completely: https://buildbot.python.org/all/#/builders/40/builds/3012 The worker process has been killed (line 562), but regrtest is still waiting. -- ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Crap, test_regrtest.test_multiprocessing_timeout() hangs sometimes (randomly), and then support.temp_cwd() fails on rmtree() with "PermissionError: [WinError 32] The process cannot access the file because it is being used by another process". running:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Eryk Sun > Alternatively, instead of special casing the file type and spinning on > PeekNamedPipe, the workaround could be based on a multiple-object wait that > includes the child process handle. (...) Well, maybe we could do something to enhance

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Eryk Sun added the comment: Alternatively, instead of special casing the file type and spinning on PeekNamedPipe, the workaround could be based on a multiple-object wait that includes the child process handle. In this case, communicate() would always call _communicate in Windows, regardless

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 767434c39c8f3c6a8af1b3282d8382ccf809fe21 by Victor Stinner in branch 'master': bpo-37531: Fix regrtest _timedout() function on timeout (GH-15419) https://github.com/python/cpython/commit/767434c39c8f3c6a8af1b3282d8382ccf809fe21 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15119 pull_request: https://github.com/python/cpython/pull/15419 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Eryk Sun added the comment: > Is there a way to workaround that? For Windows, subprocess could have a _read_all(file) method that special cases a pipe. The read loop for a pipe would check whether the child has exited. Then call _winapi.PeekNamedPipe on the handle (from get_osfhandle), and

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: > This is an issue when the standard handles are inherited or duplicated to a > grandchild process, and so on. In the case of Popen(sys.executable), the > system is duplicating the standard handles implicitly because sys.executable > is a console process

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Eryk Sun added the comment: > It seems like self.stdout.read() hangs even after the child process > has been killed. This is an issue when the standard handles are inherited or duplicated to a grandchild process, and so on. In the case of Popen(sys.executable), the system is duplicating

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: The issue can be reproduced with regrtest: * Copy attached test_kill.py to Lib/test/ * Run: python -m test test_kill -v -j2 --timeout=5 I also applied the following changes to regrtest: diff --git a/Lib/test/libregrtest/runtest.py

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Oh, I found an issue with regrtest and Windows processes. On Windows, when a process dies (crash or whatever), Popen.communicate() continues to hang in the parent if the died process has at least one child process which is still alive. Test attached

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: I happened to catch a stuck build prior to the process being killed: https://buildbot.python.org/all/#/builders//builds/2887 In short, the PR doesn't change the problem. The regrtest main will wait indefinitely on the successfully killed process. I have

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: > x86 Windows7 3.x > https://buildbot.python.org/all/#/builders/58/builds/2889 This buildbot looks slow: the 3 latest builds (2886, 2887, 2888) took around 1 hour. Example: 10 slowest tests: - test_multiprocessing_spawn: 15 min 31 sec - test_zipfile: 8 min

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Oh, interesting: test_regrtest.test_multiprocessing_timeout() failed with a timeout on x86 Windows7 3.x. This build tested commit d33e46d17d33f9b918846982c02ddc17d897c9bc which is older than my "bpo-37531: Enhance regrtest multiprocess timeout (GH-15345)"

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset de2d9eed8bc628533e1628b843cc4c7a5010f6e5 by Victor Stinner in branch 'master': bpo-37531: Enhance regrtest multiprocess timeout (GH-15345) https://github.com/python/cpython/commit/de2d9eed8bc628533e1628b843cc4c7a5010f6e5 --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: > typepref.exe (load for test_concurrent_futures) I don't understand why you get 2 typepref.exe process: only the main process should spawn one. Worker processes don't need it and should not spawn it. -- ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: Additional logging on failures is always welcome. Might I suggest that, in this case, regrtest treats this action as a hard fail so as to not got lost in the other transient failures (test_asyncio). By that I mean, either to not re-run, or to still be

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: I wrote PR 15345 to enhance regrtest timeout: more logs, more timeouts. -- ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15062 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/15345 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: (I reopen the issue.) "There are no child processes of the main test process. There are 5 orphaned processes from multiprocessing. Their parent process is gone (assuming the parent_pid argument is correct)." Oh wow, that's really strange. I would expect

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: > Can you check the process hierarchy? I would like to know how many worker > processes are still running under the main regrtest process. I expect to see > exactly one. I don't know how to investigate more on such issue on Windows. There are no child

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: > I can leave it "stuck" if there is some diagnostics that would be beneficial. Can you check the process hierarchy? I would like to know how many worker processes are still running under the main regrtest process. I expect to see exactly one. I don't know

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: I just tested the 3.7 branch manually: regrtest works as expected. I'm using this local changes: diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index e7dce180cb..7d996d6918 100644 --- a/Lib/test/libregrtest/runtest.py +++

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Jeremy Kloth added the comment: It seems that an issue still exists. https://buildbot.python.org/all/#/builders/130/builds/1050 has been running for nearly 9 hours at this point. I can leave it "stuck" if there is some diagnostics that would be beneficial. But I would prefer not to have

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- components: +Tests resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.7, Python 3.8 ___ Python tracker

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: Thanks Joannah! That's a nice enhancement which should help to stabilize buildbots even more. The next step is bpo-37711: "regrtest: re-run failed tests in subprocesses". -- ___ Python tracker

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset d85c5670ff1611202a25c9e0967148e72c114de9 by Victor Stinner in branch '3.8': [3.8] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15279) https://github.com/python/cpython/commit/d85c5670ff1611202a25c9e0967148e72c114de9

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset 93bee6a8d878ff952e245614c567c7c6bb7a0398 by Victor Stinner in branch '3.7': [3.7] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15280) https://github.com/python/cpython/commit/93bee6a8d878ff952e245614c567c7c6bb7a0398

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15004 pull_request: https://github.com/python/cpython/pull/15280 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +15002 pull_request: https://github.com/python/cpython/pull/15279 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

STINNER Victor added the comment: New changeset b0c8369c603633f445ccbb5ca7a8742145ff9eec by Victor Stinner in branch 'master': bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) https://github.com/python/cpython/commit/b0c8369c603633f445ccbb5ca7a8742145ff9eec --

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by STINNER Victor : -- pull_requests: +14996 pull_request: https://github.com/python/cpython/pull/15072 ___ Python tracker ___

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by Jeremy Kloth : -- nosy: +jkloth ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

Change by Joannah Nanjekye : -- keywords: +patch pull_requests: +14491 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/14679 ___ Python tracker

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

New submission from Joannah Nanjekye : Reported by Victor Stinner. regrtest has a --timeout parameter which calls faulthandler.dump_traceback_later(timeout), but sometimes it's not enough to kill a test. regrtest should kill a worker process if it runs longer than --timeout seconds. *