[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Indeed, it seems this should only be disabled when the "fork" model is used, 
especially as the optimization is mostly valuable when spawning a worker is 
expensive.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Gregory P. Smith]

Gregory P. Smith  added the comment:

and similarly, the dynamic spawning could be kept when the mp_context is spawn 
(and possibly forkserver).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Gregory P. Smith]

Gregory P. Smith  added the comment:

It sounds like we need to introspect the mp_context= passed to 
ProcessPoolExecutor (and it's default when None) to raise an error when 
max_tasks_per_child is incompatible with it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Inada Naoki]

Inada Naoki  added the comment:

> The only way to safely launch worker processes on demand is to spawn a worker 
> launcher process spawned prior to any thread creation that remains idle, with 
> a sole job of spawn new worker processes for us. That sounds complicated. 
> That'd be a feature. Lets go with the bugfix first.

fork is not the only way to launch worker process. We have spawn. And sapwn is 
the default for macOS since Python 3.8.

Simple reverting seems not good for macOS users, since they need to pay cost 
for both of pre-spawning and spawn.
Can't we just pre-spawn only when fork is used?

--
nosy: +methane

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Gregory P. Smith]

Change by Gregory P. Smith :


--
keywords: +patch
pull_requests: +29029
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/30847

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Gregory P. Smith]

Gregory P. Smith  added the comment:

https://bugs.python.org/issue44733 for 3.11 attempts to build upon the dynamic 
spawning ability and will need reverting unless a non-thread+fork 
implementation is provided.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Gregory P. Smith]

Gregory P. Smith  added the comment:

For context, the fundamental problem is that os.fork() is unsafe to use in any 
process with threads.  concurrent/futures/process.py violates this.  So long as 
its design involves a thread, it can never be guaranteed not to deadlock.

POSIX forbids execution of most code after fork() has been called.  Returning 
to the CPython interpreter after os.fork() in the child process is never safe 
unless the parent process had no threads at fork time.

The change that triggered the issue moved from doing all of the os.fork() calls 
for the workers up front *before the thread was spawned* to doing them on 
demand after the thread was running.

The bugfix for this is simply a rollback to revert the bpo-39207 change. It was 
a performance enhancement, but it causes a new deadlock in code that didn't 
already have one when thread tuned malloc implementations are used.

The only way to safely launch worker processes on demand is to spawn a worker 
launcher process spawned prior to any thread creation that remains idle, with a 
sole job of spawn new worker processes for us. That sounds complicated. That'd 
be a feature. Lets go with the bugfix first.

--
assignee:  -> gregory.p.smith
keywords: +3.9regression
nosy: +gregory.p.smith
stage:  -> needs patch
type:  -> resource usage

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[Yilei Yang]

New submission from Yilei Yang :

When Python is built and linked with tcmalloc, using ProcessPoolExecutor may 
deadlock. Here is a reproducible example:

$ cat t.py
from concurrent import futures
import sys
def work(iteration, item):
  sys.stdout.write(f'working: iteration={iteration}, item={item}\n')
  sys.stdout.flush()
for i in range(0, 1):
with futures.ProcessPoolExecutor(max_workers=2) as executor:
executor.submit(work, i, 1)
executor.submit(work, i, 2)

$ python t.py
working: iteration=0, item=1
working: iteration=0, item=2
working: iteration=1, item=1
working: iteration=1, item=2
...
working: iteration=3631, item=1
working: iteration=3631, item=2


The child process fails to finish. It's more likely to reproduce when the 
system is busy.

With some bisect search internally, this commit 
https://github.com/python/cpython/commit/1ac6e379297cc1cf8acf6c1b011fccc7b3da2cbe
 "triggered" the deadlock threshold with tcmalloc.

--
components: Library (Lib)
messages: 411208
nosy: yilei
priority: normal
severity: normal
status: open
title: concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
versions: Python 3.10, Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com