[issue31886] Multiprocessing.Pool hangs after re-spawning several worker process.

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +Olivier.Grisel, tomMoral

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31886] Multiprocessing.Pool hangs after re-spawning several worker process.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Generally speaking, queues can remain in an inconsistent state after a process 
crash (because the process might have crashed just after acquiring a shared 
semaphore or sending part of a large message).  It's not obvious to me how we 
could make them safer, at least under Unix where there's no widely-available 
message-oriented communication facility that I know of.

--
nosy: +davin, pitrou
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31886] Multiprocessing.Pool hangs after re-spawning several worker process.

[olarn]

New submission from olarn :

Multiprocessing's pool apparently attempts to repopulate the pool in an event 
of sub-process worker crash. However the pool seems to hangs after about ~ 
4*(number of worker) process re-spawns.

I've tracked the issue down to queue.get() stalling at multiprocessing.pool, 
line 102

Is this a known issue? Are there any known workaround?

To reproduce this issue:

import multiprocessing
import multiprocessing.util
import logging

multiprocessing.util._logger = multiprocessing.util.log_to_stderr(logging.DEBUG)
import time
import ctypes


def crash_py_interpreter():
print("attempting to crash the interpreter in ", 
multiprocessing.current_process())
i = ctypes.c_char('a'.encode())
j = ctypes.pointer(i)
c = 0
while True:
j[c] = 'a'.encode()
c += 1
j


def test_fn(x):
print("test_fn in ", multiprocessing.current_process().name, x)
exit(0)

time.sleep(0.1)


if __name__ == '__main__':

# pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
pool = multiprocessing.Pool(processes=1)

args_queue = [n for n in range(20)]

# subprocess quits
pool.map(test_fn, args_queue)

# subprocess crashes
# pool.map(test_fn,queue)

--
components: Library (Lib)
messages: 305124
nosy: olarn
priority: normal
severity: normal
status: open
title: Multiprocessing.Pool hangs after re-spawning several worker process.
type: behavior
versions: Python 2.7, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com