[issue39751] multiprocessing breaks when payload fails to unpickle

[Irit Katriel]

Irit Katriel  added the comment:

Changing type since crash typically means segfault and not an exception.

--
nosy: +iritkatriel
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39751] multiprocessing breaks when payload fails to unpickle

[João Eiras]

Change by João Eiras :


--
type:  -> crash

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39751] multiprocessing breaks when payload fails to unpickle

[João Eiras]

New submission from João Eiras :

The multiprocessing module uses pickles to send data between processes.

If a blob fails to unpickle (bad implementation of __setstate__, invalid 
payload from __reduce__, random crash in __init__) when the multiprocessing 
module will crash inside the _handle_results worker, e.g.:

  File "lib\threading.py", line 932, in _bootstrap_inner
self.run()
  File "lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
  File "lib\multiprocessing\pool.py", line 576, in _handle_results
task = get()
  File "lib\multiprocessing\connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
  TypeError: __init__() takes 1 positional argument but 4 were given

After this the worker has crashed and every task waiting from results from the 
pool will wait forever.

There are 2 things that I think should be fixed:
1. in handle_results, capture all unrecognized errors and propagate in the main 
thread. At this point at least one of the jobs' replies is lost forever so 
there is little point in trying to log and resume.
2. separate the result payload from the payload that contains the job index/id 
so they are unpickled in two steps. The first step unpickles the data internal 
to multiprocessing to know which task the result refers to. The second step 
unpickles the return value or exception from the function that was called, and 
if this object fails to unpickle, propagate that error to the main thread 
through the proper ApplyResult or IMapIterator instances.

--
components: email
files: test_multiproc_error_unpickle.py
messages: 362649
nosy: João Eiras, barry, r.david.murray
priority: normal
severity: normal
status: open
title: multiprocessing breaks when payload fails to unpickle
versions: Python 3.8
Added file: https://bugs.python.org/file48915/test_multiproc_error_unpickle.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com