New submission from STINNER Victor <[email protected]>:
Currently, multiprocessing.Pool._worker_handler() checks every 100 ms if a
worker exited using time.sleep(0.1). It causes a latency if worker exit
frequently and the pool has to execute a large number of tasks.
Worst case:
---
import multiprocessing
import time
CONCURRENCY = 1
NTASK = 100
def noop():
pass
with multiprocessing.Pool(CONCURRENCY, maxtasksperchild=1) as pool:
start_time = time.monotonic()
results = [pool.apply_async(noop, ()) for _ in range(NTASK)]
for result in results:
result.get()
dt = time.monotonic() - start_time
pool.terminate()
pool.join()
print("Total: %.1f sec" % dt)
---
Output:
---
Total: 10.2 sec
---
The worst case is a pool of 1 process, each worker only executes a single task
and the task does nothing (minimize task execution time): the latency is 100 ms
per task, which means 10 seconds for 100 tasks.
Using SIGCHLD signal to be notified when a worker completes would allow to
avoid polling: reduce the latency and reduce CPU usage (the thread doesn't have
to be awaken every 100 ms anymore).
----------
components: Library (Lib)
messages: 331797
nosy: davin, pablogsal, pitrou, vstinner
priority: normal
severity: normal
status: open
title: multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on
worker exit
versions: Python 3.8
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue35493>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
