BinbinLiang created ARROW-10676:
-----------------------------------
Summary: [Python] pickle error occurs when using
pyarrow._plasma.PlasmaClient with multiprocess on mac (python3.8.5)
Key: ARROW-10676
URL: https://issues.apache.org/jira/browse/ARROW-10676
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 2.0.0, 0.17.1
Environment: OS: mac os catalina 10.15.6;
Python version: python3.8.5;
Reporter: BinbinLiang
* The environment is：
** OS: mac os catalina 10.15.6;
** {color:#FF0000}Python version: python3.8.5;{color}
* {color:#ff0000}It's ok, when use python3.7(3.7.2 or 3.7.3) or
python3.6(3.6.8).
The error occurs only in python3.8.5.{color}
* When I use 'pyarrow._plasma.PlasmaClient' in the
'multiprocessing.context.Process', the error occurs.
** First, I have a subclass of 'multiprocessing.context.Process', named
Executor;
The defined code like this: 'class Executor(Process):' .
** And then, I create a 'PlasmaClient' object in the '__init__' function of
'class Executor' .
** Finally, I new a 'Executor' object, and call the 'start()' function.
* The detail informations of traceback are as follows:
Traceback (most recent call last):
File
"/Users/liangbinbin/PycharmProjects/waimai_data_cubeinsight/python/taf/server/taf_server.py",
line 100, in LaunchExecutor
executor.start()
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/process.py",
line 121, in start
self._popen = self._Popen(self)
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/context.py",
line 284, in _Popen
return Popen(process_obj)
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/popen_spawn_posix.py",
line 32, in __init__
super().__init__(process_obj)
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/popen_fork.py",
line 19, in __init__
self._launch(process_obj)
File
"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/popen_spawn_posix.py",
line 47, in _launch
reduction.dump(process_obj, fp)
File
{color:#FF0000}"/Users/liangbinbin/Applications/anaconda3/envs/python3_8/lib/python3.8/multiprocessing/reduction.py",
line 60, in dump{color}
{color:#FF0000}ForkingPickler(file, protocol).dump(obj){color}
{color:#FF0000}File "stringsource", line 2, in
*pyarrow._plasma.PlasmaClient.__reduce_cython__*{color}
TypeError: no default __reduce__ due to non-trivial __cinit__
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
