New submission from Philipp Lies:
I just stumbled upon a very serious bug in cPickle where cPickle stores the
data passed to it only partially without a warning/error:
#creating a 8GB long random data sting
import os
import cPickle
random_string = os.urandom(int(1.1*2**33))
print len
Philipp Lies p...@bethgelab.org added the comment:
a) it's 122GB free RAM (out of 128GB total RAM)
b) when I convert the numpy array to a list it works. So seems to be a problem
with cPickle and numpy at/from a certain array size
c) $ /usr/bin/time -v python test_np.py
Traceback (most recent
Philipp Lies p...@bethgelab.org added the comment:
Well, replace cPickle by pickle and it works. So if there is a memory
allocation problem cPickle should be able to handle it, especially since it
should be completely compatible to pickle
