[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Arkadiusz Miśkiewicz]

Arkadiusz Miśkiewicz  added the comment:

And also https://bugs.python.org/issue38744 on Linux and FreeBSD

--
nosy: +arekm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Stefano Rivera]

Stefano Rivera  added the comment:

This change seems to be causing a deadlock in multiprocessing shut-down: 
bpo-38501

--
nosy: +stefanor

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 7c994549dcffd0d9d3bb37475e6374f356e7240e by Pablo Galindo in 
branch 'master':
bpo-35493: Use Process.sentinel instead of sleeping for polling worker status 
in multiprocessing.Pool (#11488)
https://github.com/python/cpython/commit/7c994549dcffd0d9d3bb37475e6374f356e7240e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch, patch
pull_requests: +11001, 11002
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +11001
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch, patch, patch
pull_requests: +11001, 11002, 11003
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Antoine Pitrou]

Antoine Pitrou  added the comment:

If using Process.sentinel looks easy enough, then go for it.  Existing users of 
multiprocessing.Pool will benefit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

@Antoine Do you think we should start planning one of these long term solutions 
or we should start trying to use Process.sentinel as a short term solution for 
this particular issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I have to investigate how Process.sentinel can be used here.

Look how concurrent.futures uses it:
https://github.com/python/cpython/blob/master/Lib/concurrent/futures/process.py#L348

This also means:
1) we could redirect people to ProcessPoolExecutor instead of trying to 
backport all its features into multiprocessing.Pool
2) we could try to refactor the ProcessPoolExecutor implementation into a 
common backend for both ProcessPoolExecutor and multiprocessing.Pool

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Using asyncio internally would be an interesting long-term goal, at least for 
the process pool version.

Perhaps a first step is to find out how to await a multiprocessing Connection 
or Queue, or make async versions of these classes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[STINNER Victor]

STINNER Victor  added the comment:

> How do you use SIGCHLD on Windows?

I'm only proposing to use a signal when it's available, on UNIX. So have 
multiple implementations of the function, depending on the ability to get 
notified on completion without polling.

On Windows, maybe we could use a dedicated thread to set an event once 
WaitForSingleObject/WaitForMultipleObjects completes?

The design of my bpo-35479 change is to replace polling with one or multiple 
events. Maybe we can use an event to wakeup _worker_handler() when something 
happens, but have different wants to signal this event.

I have to investigate how Process.sentinel can be used here.

I might be interesting to use asyncio internally, but I'm not sure if it's 
possible ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[Antoine Pitrou]

Antoine Pitrou  added the comment:

How do you use SIGCHLD on Windows?

There is actually a portable (and robust) solution: use Process.sentinel
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.sentinel

There is another issue: Pool is currently subclassed by ThreadPool. You'll 
probably have to make the two implementations diverge a bit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-35479: multiprocessing.Pool.join() always takes at least 100 ms.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[STINNER Victor]

STINNER Victor  added the comment:

asyncio uses SIGCHLD signal to be notified when a child process completes. 
SafeChildWatcher calls os.waitpid(pid, os.WNOHANG) on each child process, 
whereas FastChildWatcher() uses os.waitpid(-1, os.WNOHANG).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

[STINNER Victor]

New submission from STINNER Victor :

Currently, multiprocessing.Pool._worker_handler() checks every 100 ms if a 
worker exited using time.sleep(0.1). It causes a latency if worker exit 
frequently and the pool has to execute a large number of tasks.

Worst case:
---
import multiprocessing
import time
CONCURRENCY = 1
NTASK = 100
def noop():
pass
with multiprocessing.Pool(CONCURRENCY, maxtasksperchild=1) as pool:
start_time = time.monotonic()
results = [pool.apply_async(noop, ()) for _ in range(NTASK)]
for result in results:
result.get()
dt = time.monotonic() - start_time
pool.terminate()
pool.join()
print("Total: %.1f sec" % dt)
---

Output:
---
Total: 10.2 sec
---

The worst case is a pool of 1 process, each worker only executes a single task 
and the task does nothing (minimize task execution time): the latency is 100 ms 
per task, which means 10 seconds for 100 tasks.

Using SIGCHLD signal to be notified when a worker completes would allow to 
avoid polling: reduce the latency and reduce CPU usage (the thread doesn't have 
to be awaken every 100 ms anymore).

--
components: Library (Lib)
messages: 331797
nosy: davin, pablogsal, pitrou, vstinner
priority: normal
severity: normal
status: open
title: multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on 
worker exit
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com