New submission from mrjbq7:
I ran into a problem using multiprocessing to create large data objects (in
this case numpy float64 arrays with 90,000 columns and 5,000 rows) and return
them to the original python process.
It breaks in both Python 2.7 and 3.3, using numpy 1.7.0 (but with different
error messages).
It is possible the array is too large to be serialized (450 million 64-bit
numbers exceeds a 32-bit limit)?
Python 2.7
==========
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 99, in worker
put((job, i, result))
File "/usr/lib/python2.7/multiprocessing/queues.py", line 390, in put
return send(obj)
SystemError: NULL result without error in PyObject_Call
Python 3.3
==========
Traceback (most recent call last):
File "multi.py", line 18, in <module>
results = pool.map_async(make_data, range(5)).get(9999999)
File "/usr/lib/python3.3/multiprocessing/pool.py", line 562, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[array([[
0.74628629, 0.36130663, -0.65984794, ..., -0.70921838,
0.34389663, -1.7135126 ],
[ 0.60266867, -0.40652402, -1.31590562, ..., 1.44896246,
-0.3922366 , -0.85012842],
[ 0.59629641, -0.00623001, -0.12914128, ..., 0.99925511,
-2.30418136, 1.73414009],
...,
[ 0.24246639, 0.87519509, 0.24109069, ..., -0.48870107,
-0.20910332, 0.11749621],
[ 0.62108937, -0.86217542, -0.47357384, ..., 1.59872243,
0.76639995, -0.56711461],
[ 0.90976471, 1.73566475, -0.18191821, ..., 0.19784432,
-0.29741643, -1.46375835]])]'. Reason: 'error("'i' format requires
-2147483648 <= number <= 2147483647",)'
----------
files: multi.py
messages: 185344
nosy: mrjbq7
priority: normal
severity: normal
status: open
title: problem using multiprocessing with really big objects?
versions: Python 3.3
Added file: http://bugs.python.org/file29595/multi.py
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue17560>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
