[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: Thanks for closing up the issue, Victor :) -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: "test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor" I didn't see this failure recently, I close the issue. Since changes were pushed, I mark the issue as fixed. If someone has ideas

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Thomas Moreau added the comment: I did GH 19788 with a few modifications. There is only one lock that seems to mater for the perf, and I actually added one other (the one in _python_exit, which necessitate another bug fix for fork context). I did not benchmark to see if it was worth it in

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by Thomas Moreau : -- pull_requests: +19111 pull_request: https://github.com/python/cpython/pull/19788 ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Thomas Moreau added the comment: I think this is a reasonable way to move on.Some of the locks can probably be removed but this needs careful investigation and in the mean time, it hinders everyone. Thanks victor for the fast fix up! To me, an interesting observation is that the failure

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Antoine Pitrou added the comment: I looked at the change and it seemed ok to me. Perhaps Thomas can give it a look too. -- assignee: aeros -> vstinner ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: Also, thanks to Victor for coming up with an interim solution, and for the feedback from Antoine and Thomas. GH-19760 is a significant improvement from my initial proposal in GH-19751. -- ___ Python tracker

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: I am a bit concerned about the performance implication of accessing the thread wakeup through a lock in the call queue, but for now I think it's reasonable to address the buildbot failure with a locking solution while we try to find a better one. I'm not

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by Dong-hee Na : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: I'm still getting more and more buildbot emails about test_concurrent_futures, so I merged my PR 19760 to fix buildbots. Please revert or modify my PR 19760 if you have a better approach, but please check that test_killed_child() and

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: New changeset a4dfe8ede5a37576e17035dccfe109ba7752237e by Victor Stinner in branch 'master': bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760) https://github.com/python/cpython/commit/a4dfe8ede5a37576e17035dccfe109ba7752237e --

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: > With my msg367463 patch (add sleep), test_cancel_futures() fails. The test uses sleep() as a synchronization primitive: executor.submit(time.sleep, .1). That's bad, but it doesn't *have to* be fixed now. My msg367463 patch adds an artifical sleep: the

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: Thomas Moreau: "One solution would be to use the `self._shutdown_lock` from the executor to protect the call to `close` in `terminate_broken` and the call to `self._thread_wakeup.wakeup` in `shutdown`. That way, the lock is only acquired at critical points

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: With my msg367463 patch (add sleep), test_cancel_futures() fails. Example: == FAIL: test_cancel_futures (test.test_concurrent_futures.ProcessPoolForkProcessPoolShutdownTest)

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by STINNER Victor : -- pull_requests: +19082 pull_request: https://github.com/python/cpython/pull/19760 ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Antoine Pitrou added the comment: With the same traceback and error message? -- ___ Python tracker ___ ___ Python-bugs-list

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: Antoine Pitrou: "How about the following (untested): (...)" Using Antoine's patch, test_killed_child() still fails (I used my msg367463 patch to make the failure more likely). -- ___ Python tracker

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by STINNER Victor : -- pull_requests: +19080 pull_request: https://github.com/python/cpython/pull/19758 ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: I decided to close PR-19751. Both because it does not correctly address the race condition (due to an oversight on my part) and it would add substantial overhead to _ThreadWakeup. Instead, I agree that we should explore a non-locking solution. --

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: Oops, it seems that I opened PR-19751 a bit preemptively. When I get the chance, I'll see if Antoine's implementation can address the failures and do some comparisons. -- ___ Python tracker

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by Kyle Stanley : -- pull_requests: +19072 pull_request: https://github.com/python/cpython/pull/19751 ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Antoine Pitrou added the comment: How about the following (untested): diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 8e9b69a8f0..c0c2eb3032 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -66,23 +66,29 @@

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Thomas Moreau added the comment: Sorry I just saw this. It seems that I introduced this regression. One of the goal of having a `ThreadWakeup` and not a `SimpleQueue` is to avoid using locks that can hinder the performance of the executor. I don't remember the exact details but I think I

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: After writing the above out and a bit of further consideration, I think it might make more sense to wait for the event after setting `self._closed = True` so that it prevents future wakeup() and clear() calls from reading/writing to the pipe, while still

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Kyle Stanley added the comment: I can't be certain for the other failures, but I'm currently exploring a potential solution for addressing the `test_killed_child` failure. To me, it seems to be a race condition with attempting to call _ThreadWakeup.close() while there are still bytes being

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by Kyle Stanley : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by Kyle Stanley : -- nosy: +aeros ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: It seems like test_killed_child() race condition was introduced by: commit a5cbab552d294d99fde864306632d7e511a75d3c (refs/bisect/bad) Author: Thomas Moreau Date: Sun Feb 16 19:09:26 2020 +0100 bpo-39104: Fix hanging ProcessPoolExecutor on shutdown

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: > ERROR: test_killed_child > (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) The patch below makes this test failure more likely: diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py index

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: terminate_broken() method was added by: commit 0e89076247580ba0e570c4816f0e5628a7e36e83 Author: Thomas Moreau Date: Sun Mar 1 21:49:14 2020 +0100 bpo-39678: refactor queue manager thread (GH-18551) -- ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: > It seems like Connection.close() was called while Connection._send() was > called. I added debug logs: The connection was closed by terminate_broken() called by _ExecutorManagerThread.run() thread: test_killed_child

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: ERROR: test_killed_child (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) (...) File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.9/multiprocessing/connection.py", line 373,

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: x86 Gentoo Installed with X 3.x: https://buildbot.python.org/all/#/builders/128/builds/726 test_del_shutdown (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest) ... Warning -- Unraisable exception Exception ignored in: .weakref_cb at

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: AMD64 Fedora Stable Clang Installed 3.x: https://buildbot.python.org/all/#/builders/127/builds/679 0:04:21 load avg: 1.29 [423/423/1] test_concurrent_futures failed (2 min 39 sec) Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0,

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: > See also bpo-30966 "Add multiprocessing.SimpleQueue.close()". I pushed a commit 1a275013d1ecc2e3778d64fda86174b2f13d6969: "Process.shutdown(wait=True) of concurrent.futures now closes explicitly the result queue." test_shutdown_deadlock_pickle() still

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: AMD64 Ubuntu Shared 3.x: https://buildbot.python.org/all/#/builders/101/builds/809 == ERROR: test_shutdown_no_wait (test.test_concurrent_futures.ProcessPoolForkserverProcessPoolShutdownTest)

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: New changeset 5d1f32d33ba24d0aa87235ae40207bb57778388b by Victor Stinner in branch 'master': bpo-39995: Split test_concurrent_futures.test_crash() into sub-tests (GH-19739) https://github.com/python/cpython/commit/5d1f32d33ba24d0aa87235ae40207bb57778388b

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by STINNER Victor : -- keywords: +patch pull_requests: +19061 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19739 ___ Python tracker ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: See also bpo-30966 "Add multiprocessing.SimpleQueue.close()". -- ___ Python tracker ___ ___

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: > Same bug on AMD64 FreeBSD Non-Debug 3.x: > https://buildbot.python.org/all/#/builders/214/builds/472 Oh, test_crash failed twice, but not on the same test case: * ERROR: test_crash (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest) [crash

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

STINNER Victor added the comment: Same bug on AMD64 FreeBSD Non-Debug 3.x: https://buildbot.python.org/all/#/builders/214/builds/472 == ERROR: test_crash (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest)

[issue39995] test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor

Change by STINNER Victor : -- title: test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with -> test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest.test_crash() fails with OSError: [Errno 9] Bad file descriptor