[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Antoine Pitrou added the comment: Indeed, it seems this should only be disabled when the "fork" model is used, especially as the optimization is mostly valuable when spawning a worker is expensive. -- nosy: +pitrou ___ Python tracker

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Gregory P. Smith added the comment: and similarly, the dynamic spawning could be kept when the mp_context is spawn (and possibly forkserver). -- ___ Python tracker ___

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Gregory P. Smith added the comment: It sounds like we need to introspect the mp_context= passed to ProcessPoolExecutor (and it's default when None) to raise an error when max_tasks_per_child is incompatible with it. -- ___ Python tracker

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Inada Naoki added the comment: > The only way to safely launch worker processes on demand is to spawn a worker > launcher process spawned prior to any thread creation that remains idle, with > a sole job of spawn new worker processes for us. That sounds complicated. > That'd be a feature.

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Change by Gregory P. Smith : -- keywords: +patch pull_requests: +29029 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30847 ___ Python tracker

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Gregory P. Smith added the comment: https://bugs.python.org/issue44733 for 3.11 attempts to build upon the dynamic spawning ability and will need reverting unless a non-thread+fork implementation is provided. -- ___ Python tracker

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

Gregory P. Smith added the comment: For context, the fundamental problem is that os.fork() is unsafe to use in any process with threads. concurrent/futures/process.py violates this. So long as its design involves a thread, it can never be guaranteed not to deadlock. POSIX forbids

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

New submission from Yilei Yang : When Python is built and linked with tcmalloc, using ProcessPoolExecutor may deadlock. Here is a reproducible example: $ cat t.py from concurrent import futures import sys def work(iteration, item): sys.stdout.write(f'working: iteration={iteration},