[issue13555] cPickle MemoryError when loading large file (while pickle works)
Roundup Robot added the comment: New changeset f3f23ecdb1c6 by Serhiy Storchaka in branch '2.7': Issue #13555: Fix an integer overflow check. http://hg.python.org/cpython/rev/f3f23ecdb1c6 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: Thank you for the report. Standard tests do not cover pickling/unpickling to real files. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: I have opened issue17299 for testing issue. -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Arfrever Frehtes Taifersar Arahesis added the comment: 680959a3ae2e has caused that g-ir-scanner tool from gobject-introspection (which seems to create some pickles and next load them?) fails with MemoryError, e.g. during building of GTK+. MemoryError occurs only for a subset of pickles. I attach a small pickle, which allows to reproduce this problem. Download this pickle and save it in /tmp. Additional preparation: $ cd /tmp $ wget -q http://ftp.gnome.org/pub/gnome/sources/gobject-introspection/1.34/gobject-introspection-1.34.2.tar.xz $ tar -xJf gobject-introspection-1.34.2.tar.xz $ sed -e '/^with LibtoolImporter/,/^$/d' -i gobject-introspection-1.34.2/giscanner/xmlwriter.py Behavior before 680959a3ae2e: $ PYTHONPATH=gobject-introspection-1.34.2 python2.7 -c 'import cPickle; print(cPickle.load(open(pickle)))' giscanner.girparser.GIRParser object at 0x7f506299be90 Behavior after 680959a3ae2e: $ PYTHONPATH=gobject-introspection-1.34.2 python2.7 -c 'import cPickle; print(cPickle.load(open(pickle)))' Traceback (most recent call last): File string, line 1, in module MemoryError -- nosy: +benjamin.peterson priority: normal - release blocker resolution: fixed - stage: committed/rejected - status: closed - open Added file: http://bugs.python.org/file29239/pickle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Serhiy Storchaka storch...@gmail.com: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Roundup Robot added the comment: New changeset 680959a3ae2e by Serhiy Storchaka in branch '2.7': Issue #13555: cPickle now supports files larger than 2 GiB. http://hg.python.org/cpython/rev/680959a3ae2e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file29020/pickle_overflow-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: Test updated too. Now it doesn't try to write a string larger than 2 GiB (it's impossible), instead writes a lot of shorter strings with total size larger than 2 GiB. -- Added file: http://bugs.python.org/file29042/pickle_overflow-4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: Here is an updated patch. Fixed two bugs found by Antoine (an inappropriate format and a memory error in bigmemtest), fixed resizing of marks array and one possible integer overflow in write_other(). Used workaround to bypass limitations of cStringIO API. -- assignee: - serhiy.storchaka stage: needs patch - patch review Added file: http://bugs.python.org/file29020/pickle_overflow-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Serhiy Storchaka storch...@gmail.com: -- dependencies: +cStringIO not 64-bit safe -cStringIO.StringIO aborted when more then INT_MAX bytes written ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali added the comment: Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments. No. I don't have access to a 64-bit box, which makes it difficult to write (I don't get compiler warnings) and test. Feel free to pick up the patch and update it! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Serhiy Storchaka added the comment: I have picked up it. Actually BINSTRING format supports only strings less 2GiB, so test should be changed. But there are other bugs which prohibit pickling a lot of data. For one I open issue17054. -- dependencies: +cStringIO.StringIO aborted when more then INT_MAX bytes written ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Ramchandra Apte added the comment: Bump. @neologix I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou added the comment: I may tackle this but rare 2.7-only bugs are pretty low on my priorities list. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali added the comment: I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap) AFAICT, a binary string a little longer than 1GB should be enough to reproduce the bug. Just make sure Python isn't built with '-fwrapv'. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali added the comment: Could someone with a 64-bit box take over this one? I won't go anywhere with my 32-bit box... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: Antoine, could you test the last version (test_pickle and if possible with the OP testcase)? I can't test it myself (32-bit machine with 1 GB). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: Le vendredi 06 janvier 2012 à 19:17 +, Charles-François Natali a écrit : Charles-François Natali neolo...@free.fr added the comment: Antoine, could you test the last version (test_pickle and if possible with the OP testcase)? I can't test it myself (32-bit machine with 1 GB). Well, first, there are compilation warnings on a 64-bit box: gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/usr/local/include -I/home/antoine/cpython/27/Include -I/home/antoine/cpython/27 -c /home/antoine/cpython/27/Modules/cPickle.c -o build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o /home/antoine/cpython/27/Modules/cPickle.c: In function ‘put2’: /home/antoine/cpython/27/Modules/cPickle.c:810:9: attention : format ‘%d’ expects type ‘int’, but argument 4 has type ‘Py_ssize_t’ /home/antoine/cpython/27/Modules/cPickle.c: In function ‘load_mark’: /home/antoine/cpython/27/Modules/cPickle.c:4610:21: attention : assignment from incompatible pointer type gcc -pthread -shared build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o -L/usr/local/lib -o build/lib.linux-x86_64-2.7-pydebug/cPickle.so Second, I can't seem to get the test to work with 8GB RAM (approximately 6.5GB free according to free). The MemoryError is quite expectable for test_pickle, though, since the code there doesn't try to conserve memory at all: test test_pickle failed -- Traceback (most recent call last): File /home/antoine/cpython/27/Lib/test/test_support.py, line 983, in wrapper return f(self, maxsize) File /home/antoine/cpython/27/Lib/test/pickletester.py, line 1298, in test_huge_str_32b pickled = self.dumps(data, proto) File /home/antoine/cpython/27/Lib/test/test_pickle.py, line 74, in dumps return pickle.dumps(arg, proto) File /home/antoine/cpython/27/Lib/pickle.py, line 1374, in dumps Pickler(file, protocol).dump(obj) File /home/antoine/cpython/27/Lib/pickle.py, line 224, in dump self.save(obj) File /home/antoine/cpython/27/Lib/pickle.py, line 286, in save f(self, obj) # Call unbound method with explicit self File /home/antoine/cpython/27/Lib/pickle.py, line 488, in save_string self.write(STRING + repr(obj) + '\n') MemoryError I would therefore suggest to enable the test only for cPickle. For test_cpickle the behaviour is different: - for protocol 0, I get a MemoryError (which may be expected, if the test truly needs more than 6.5GB, for example because of suboptimal buffer management) - for protocol 1 and 2, I get SystemError: error return without exception set -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment:
Here's a new version with a test (untested).
Note that I'm absolutely not sure that the 'memsize' argument to
bigmemtest is correct.
--
Added file: http://bugs.python.org/file24079/pickle_overflow-2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -6,7 +6,8 @@
import pickletools
import copy_reg
-from test.test_support import TestFailed, have_unicode, TESTFN
+from test.test_support import (TestFailed, have_unicode, TESTFN, _2G,
+ precisionbigmemtest)
# Tests that try a number of pickle protocols should have a
# for proto in protocols:
@@ -14,6 +15,8 @@
assert pickle.HIGHEST_PROTOCOL == cPickle.HIGHEST_PROTOCOL == 2
protocols = range(pickle.HIGHEST_PROTOCOL + 1)
+ascii_char_size = 1
+
# Copy of test.test_support.run_with_locale. This is needed to support Python
# 2.4, which didn't include it. This is all to support test_xpickle, which
# bounces pickled objects through older Python versions to test backwards
@@ -1280,3 +1283,21 @@
f.write(pickled2)
f.seek(0)
self.assertEqual(unpickler.load(), data2)
+
+class BigmemPickleTests(unittest.TestCase):
+
+# All protocols use 1-byte per printable ASCII character; we add another
+# byte because the encoded form has to be copied into the internal buffer.
+
+@precisionbigmemtest(size=_2G, memuse=3 + ascii_char_size)
+def test_huge_str_32b(self, size):
+data = abcd * (size // 4)
+try:
+for proto in protocols:
+try:
+pickled = self.dumps(data, proto)
+self.loads(pickled)
+finally:
+pickled = None
+finally:
+data = None
diff --git a/Lib/test/test_cpickle.py b/Lib/test/test_cpickle.py
--- a/Lib/test/test_cpickle.py
+++ b/Lib/test/test_cpickle.py
@@ -1,7 +1,9 @@
import cPickle, unittest
from cStringIO import StringIO
-from test.pickletester import AbstractPickleTests, AbstractPickleModuleTests
-from test.pickletester import AbstractPicklerUnpicklerObjectTests
+from test.pickletester import (AbstractPickleTests,
+ AbstractPickleModuleTests,
+ AbstractPicklerUnpicklerObjectTests,
+ BigmemPickleTests)
from test import test_support
class cPickleTests(AbstractPickleTests, AbstractPickleModuleTests):
@@ -101,6 +103,16 @@
pickler_class = cPickle.Pickler
unpickler_class = cPickle.Unpickler
+class cPickleBigmemPickleTests(BigmemPickleTests):
+
+def dumps(self, arg, proto=0, fast=0):
+# Ignore fast
+return cPickle.dumps(arg, proto)
+
+def loads(self, buf):
+# Ignore fast
+return cPickle.loads(buf)
+
class Node(object):
pass
@@ -133,6 +145,7 @@
cPickleFastPicklerTests,
cPickleDeepRecursive,
cPicklePicklerUnpicklerObjectTests,
+cPickleBigmemPickleTests,
)
if __name__ == __main__:
diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py
--- a/Lib/test/test_pickle.py
+++ b/Lib/test/test_pickle.py
@@ -3,10 +3,11 @@
from test import test_support
-from test.pickletester import AbstractPickleTests
-from test.pickletester import AbstractPickleModuleTests
-from test.pickletester import AbstractPersistentPicklerTests
-from test.pickletester import AbstractPicklerUnpicklerObjectTests
+from test.pickletester import (AbstractPickleTests,
+ AbstractPickleModuleTests,
+ AbstractPersistentPicklerTests,
+ AbstractPicklerUnpicklerObjectTests,
+ BigmemPickleTests)
class PickleTests(AbstractPickleTests, AbstractPickleModuleTests):
@@ -66,6 +67,16 @@
pickler_class = pickle.Pickler
unpickler_class = pickle.Unpickler
+class PickleBigmemPickleTests(BigmemPickleTests):
+
+def dumps(self, arg, proto=0, fast=0):
+# Ignore fast
+return pickle.dumps(arg, proto)
+
+def loads(self, buf):
+# Ignore fast
+return pickle.loads(buf)
+
def test_main():
test_support.run_unittest(
@@ -73,6 +84,7 @@
PicklerTests,
PersPicklerTests,
PicklerUnpicklerObjectTests,
+PickleBigmemPickleTests,
)
test_support.run_doctest(pickle)
diff --git a/Modules/cPickle.c b/Modules/cPickle.c
--- a/Modules/cPickle.c
+++ b/Modules/cPickle.c
@@ -139,15 +139,15 @@
typedef struct {
PyObject_HEAD
-int length; /* number of initial slots in data currently used */
-int size; /* number of slots in data allocated */
+Py_ssize_t length; /* number of
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: New version. Looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment:
So it seems unlikely to be the explanation.
Victor reproduced in on IRC, and it's indeed an overflow.
The problematic code is in readline_file:
bigger = self-buf_size 1;
if (bigger = 0) { /* overflow */
PyErr_NoMemory();
return -1;
}
newbuf = (char *)realloc(self-buf, bigger);
if (!newbuf) {
PyErr_NoMemory();
return -1;
}
self-buf_size is an int, which overflow pretty easily.
196 * 24
4704
196 * 24 * 8 # assuming 8 bytes per float
37632
2**31
2147483648
Hmmm... A byte is 8 bit, which gives:
196 * 24 * 8 * 8
301056L
196 * 24 * 8 * 8 2**31
True
Now, if it works on your box, it's probably due to the compiler optimizing the
check away. Since `bigger` is cast to an unsigned 64-bit (size_t) when calling
malloc(), it happens to work.
Maybe your distro doesn't build python with -fwrapv.
So, what do you suggest? Should we fix this (Py_ssize_t, overflow check before
computation), as in #11564?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
STINNER Victor victor.stin...@haypocalc.com added the comment: Should we fix this (Py_ssize_t, overflow check before computation), as in #11564? Yes. Use Py_ssize_t type for the buf_size attribute, and replace bigger = 0 (test if an overflow occurred) by self-buf_size (PY_SSIZE_T_MAX 1). -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: Ah, I see. It's a bit of a pity not to be able to load files 1GB, especially on a 64-bit build (!). Perhaps cPickle could be made partly 64-bit compatible? Or at least, indeed, do a proper anti-overflow check. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: Here's a patch which should fix this. However, I'm unable to test it. -- keywords: +patch Added file: http://bugs.python.org/file24006/pickle_overflow.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: I think there's a problem here: +self-data = realloc(self-data, self-size * sizeof(PyObject *)); +if (self-data == NULL) goto nomemory; If realloc() fails, the old data pointer is lost and therefore will never get free()ed. Same for: +self-buf = (char *)realloc(self-buf, self-buf_size); Here: -int *marks; -s=self-marks_size+20; -if (s = self-num_marks) s=self-num_marks + 1; +size_t alloc; +Py_ssize_t *marks; + +/* Use the size_t type to check for overflow. */ +alloc = ((size_t)self-num_marks 1) + 20; It seems you are changing the overallocation algorithm (from additive to multiplicative). I'm not sure it should be in the scope of the patch, although multiplicative overallocation is generally better. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: New version. -- Added file: http://bugs.python.org/file24008/pickle_overflow-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Philipp Lies p...@bethgelab.org added the comment: a) it's 122GB free RAM (out of 128GB total RAM) b) when I convert the numpy array to a list it works. So seems to be a problem with cPickle and numpy at/from a certain array size c) $ /usr/bin/time -v python test_np.py Traceback (most recent call last): File test_np.py, line 12, in module A2 = cPickle.load(f2) MemoryError Command exited with non-zero status 1 Command being timed: python test_np.py User time (seconds): 73.72 System time (seconds): 4.56 Percent of CPU this job got: 87% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:29.52 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 7402448 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 726827 Voluntary context switches: 41043 Involuntary context switches: 7793 Swaps: 0 File system inputs: 3368 File system outputs: 2180744 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1 hth -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? AFAICT this wasn't fixed in 2.7. Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. @Philipp I'd be curious to see the last lines of $ ltrace -e malloc python test_np.py you'll probably see a call to malloc() return NULL. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. AFAICT this wasn't fixed in 2.7. @Philipp I'd be curious to see the last lines of $ ltrace -e malloc python test_np.py you'll probably see a call to malloc() return NULL. Also, depending on Antoine's answer, trying with default (3.3) could be useful. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Charles-François Natali neolo...@free.fr: -- Removed message: http://bugs.python.org/msg149321 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: Couldn't this be linked to #11564 (pickle not 64-bit ready)? Well, I don't know anything about numpy, but: 196 * 24 4704 196 * 24 * 8 # assuming 8 bytes per float 37632 2**31 2147483648 So it seems unlikely to be the explanation. After all the report says 1GB for the file size, which is still comfortably in the capabilities of a 32-bit module. Philipp, perhaps you could try to run Python under gdb and try to diagnose where the MemoryError occurs? Also, posting the disassembly (using pickletools.dis()) of a very small equivalent pickle (e.g. of randn(2,3)) could help us know what is involved. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: Oh, what's more interesting is that it works here (Python 2.7.1 and numpy 1.6.1 under Mageia with 8GB RAM). Looking at a pickle disassembly, the only remarkable thing is the presence of a long binary string (the raw serialization of all IEEE floats), which shouldn't give any problem to cPickle (and indeed doesn't). What is your numpy version, Philipp? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
New submission from Phillies p...@bethgelab.org:
When I try to load a large file (1GB) cPickle crashes with a MemoryError:
$python test.py
Traceback (most recent call last):
File /tmp/test.py, line 8, in module
A2 = cPickle.load(f2)
MemoryError
test.py contains following code:
import numpy as np
import cPickle
A = np.random.randn(196,24)
f = open('test.pydat', 'w')
cPickle.dump(A,f)
f.close()
f2 = open('test.pydat', 'rb')
A2 = cPickle.load(f2)
System:
cPickle 1.71
python 2.7.2
Ubuntu 11.10 amd64
Memory is not an issue as a) pickle works nicely and b) my computer has 122GB
free RAM
--
components: None
messages: 149027
nosy: phillies
priority: normal
severity: normal
status: open
title: cPickle MemoryError when loading large file (while pickle works)
type: crash
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
maniram maniram maniandra...@gmail.com added the comment: Maybe Ubuntu doesn't think it is safe to allocate the memory. -- nosy: +maniram.maniram ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Philipp Lies p...@bethgelab.org added the comment: Well, replace cPickle by pickle and it works. So if there is a memory allocation problem cPickle should be able to handle it, especially since it should be completely compatible to pickle. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
maniram maniram maniandra...@gmail.com added the comment: Have you checked the system monitor after all cPickle can use more memory than 1GB. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Antoine Pitrou pit...@free.fr added the comment: Is there a way to reproduce that doesn't involve numpy? -- components: +Extension Modules -None nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13555] cPickle MemoryError when loading large file (while pickle works)
Charles-François Natali neolo...@free.fr added the comment: my computer has 122GB free RAM 122, or 1.22? Well, replace cPickle by pickle and it works. cPickle makes some direct call to malloc()/realloc()/free(), contrarily to pickle which uses pymalloc. This could lead to heap fragmentation. What does the following return ? $ /usr/bin/time -v python test.py -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
