[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Arkadiusz Miśkiewicz added the comment: And also https://bugs.python.org/issue38744 on Linux and FreeBSD -- nosy: +arekm ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Stefano Rivera added the comment: This change seems to be causing a deadlock in multiprocessing shut-down: bpo-38501 -- nosy: +stefanor ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Pablo Galindo Salgado added the comment: New changeset 7c994549dcffd0d9d3bb37475e6374f356e7240e by Pablo Galindo in branch 'master': bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool (#11488)

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Change by Pablo Galindo Salgado : -- keywords: +patch, patch pull_requests: +11001, 11002 stage: -> patch review ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Change by Pablo Galindo Salgado : -- keywords: +patch pull_requests: +11001 stage: -> patch review ___ Python tracker ___ ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Change by Pablo Galindo Salgado : -- keywords: +patch, patch, patch pull_requests: +11001, 11002, 11003 stage: -> patch review ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Antoine Pitrou added the comment: If using Process.sentinel looks easy enough, then go for it. Existing users of multiprocessing.Pool will benefit. -- ___ Python tracker

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Pablo Galindo Salgado added the comment: @Antoine Do you think we should start planning one of these long term solutions or we should start trying to use Process.sentinel as a short term solution for this particular issue? -- ___ Python tracker

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Antoine Pitrou added the comment: > I have to investigate how Process.sentinel can be used here. Look how concurrent.futures uses it: https://github.com/python/cpython/blob/master/Lib/concurrent/futures/process.py#L348 This also means: 1) we could redirect people to ProcessPoolExecutor

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Antoine Pitrou added the comment: Using asyncio internally would be an interesting long-term goal, at least for the process pool version. Perhaps a first step is to find out how to await a multiprocessing Connection or Queue, or make async versions of these classes. --

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

STINNER Victor added the comment: > How do you use SIGCHLD on Windows? I'm only proposing to use a signal when it's available, on UNIX. So have multiple implementations of the function, depending on the ability to get notified on completion without polling. On Windows, maybe we could use a

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

Antoine Pitrou added the comment: How do you use SIGCHLD on Windows? There is actually a portable (and robust) solution: use Process.sentinel https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.sentinel There is another issue: Pool is currently subclassed by

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

STINNER Victor added the comment: See also bpo-35479: multiprocessing.Pool.join() always takes at least 100 ms. -- ___ Python tracker ___

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

STINNER Victor added the comment: asyncio uses SIGCHLD signal to be notified when a child process completes. SafeChildWatcher calls os.waitpid(pid, os.WNOHANG) on each child process, whereas FastChildWatcher() uses os.waitpid(-1, os.WNOHANG). --

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

New submission from STINNER Victor : Currently, multiprocessing.Pool._worker_handler() checks every 100 ms if a worker exited using time.sleep(0.1). It causes a latency if worker exit frequently and the pool has to execute a large number of tasks. Worst case: --- import multiprocessing