[issue43306] Error in multiprocessing.Pool's initializer doesn't stop execution

[Aaron]

Aaron  added the comment:

What should the behavior be if an exception is raised in a pool worker during 
bootstrapping / initialization function execution? I think an exception should 
be raised in the process owning the Pool, and in the fix I'm tinkering around 
with I just raise a RuntimeError currently. I can see an argument also for 
raising different exceptions (or having different behavior) for bootstrapping 
error vs init function, but implementation is more complicated. My current 
implementation simply creates a lock in _repopulate_pool_static, acquires it, 
and waits for the worker function to release it. By polling every 100ms I also 
detect if the process exited before releasing the lock in which case I raise a 
Runtime error. I just started testing this implementation, but I'll provide it 
for anyone else who wants to test / comment.

--
Added file: https://bugs.python.org/file50230/pool.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43306] Error in multiprocessing.Pool's initializer doesn't stop execution

[Aaron]

Aaron  added the comment:

I ran into this bug answering this question on Stack Overflow: 
https://stackoverflow.com/questions/68890437/cannot-use-result-from-multiprocess-pool-directly

I have minimized the code required to replicate the behavior, but it boils down 
to: when using "spawn" to create a multiprocessing pool, if an exception occurs 
during the bootstrapping phase of the new child or during the initialization 
function with any start method, it is just cleaned up, and another takes its 
place (which will also fail). This creates an infinite loop of creating child 
workers, workers exiting due to an exception, and re-populating the pool with 
new workers.


```
import multiprocessing
multiprocessing.set_start_method("spawn")
# bootstraping only problem with spawn

def task():
print("task")

if __name__ == "__main__":
with multiprocessing.Pool() as p:
p.apply(task)
else:
raise Exception("raise in child during bootstraping phase")

#
# or
#

import multiprocessing
# multiprocessing.set_start_method("fork")
# fork or spawn doesn't matter

def task():
print("task")

def init():
raise Exception("raise in child during initialization function")

if __name__ == "__main__":
with multiprocessing.Pool(initializer=init) as p:
p.apply(task)
```

If Pool._join_exited_workers could determine if a worker exited before 
bootstrapping, or the initialization function completed, It would indicate a 
likely significant problem. I'm fine with exceptions in the worker target 
function not being re-raised in the parent, however it seems the Pool should 
stop trying if it's failing to create new workers.

--
nosy: +athompson6735
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43306] Error in multiprocessing.Pool's initializer doesn't stop execution

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Can you add a minimal example with the ignore behavior?

--
assignee: docs@python -> 
nosy: +davin, pitrou, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43306] Error in multiprocessing.Pool's initializer doesn't stop execution

[Dávid Nemeskey]

New submission from Dávid Nemeskey :

There is an inconsistency in how multiprocessing.Pool handles exceptions thrown 
in the workers:
- exceptions raised by the mapped function stop execution right away
- exceptions raised in an initializer are ignored and the pool continues 
spawning new workers indefinitely, each of them failing.

I believe the behavior should be the same in both cases, and of the two, the 
first one is preferable (especially since that's what people are used to).

The documentation doesn't cover how exceptions are handled in pools, either.

--
assignee: docs@python
components: Documentation, Library (Lib)
messages: 387577
nosy: docs@python, nemeskeyd
priority: normal
severity: normal
status: open
title: Error in multiprocessing.Pool's initializer doesn't stop execution
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com