[issue25465] Pickle uses O(n) memory overhead

Stefan Krah added the comment: It's a Linux issue. Disable overcommitting of memory (at your own peril) or set user limits (for example with djb's softlimit), then the process will be killed instead of freezing the machine. -- nosy: +skrah ___ Python

[issue25465] Pickle uses O(n) memory overhead

Serhiy Storchaka added the comment: There is a workaround for memory consumption, and Linux freezing is not Python issue. -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker

[issue25465] Pickle uses O(n) memory overhead

Herbert added the comment: It may be fair to note that I have no swap installed on one of the machines, just 16GB of RAM, on which the 'crash' happens. Hence I'm not sure how this affects paging, I would think there is no paging if there is no swap. I can verify that the machine is 'stuck'

[issue25465] Pickle uses O(n) memory overhead

Lukas Lueg added the comment: I very strongly doubt that it actually crashes your kernel - it basically can't. Your desktop becomes unresponsive for up to several minutes as the kernel has paged out about every single bit of memory to disk, raising access times by several orders of magnitude.

[issue25465] Pickle uses O(n) memory overhead

Martin Panter added the comment: Perhaps by OS crash you mean either the Linux out-of-memory (OOM) killer, that takes a hueristic stab at killing the right process, or Linux running almost out of memory, and everything grinding to a halt presumably because each task switch needs to re-read

[issue25465] Pickle uses O(n) memory overhead

Herbert added the comment: Hi Eric, I would assume that for the right range-parameter (in my case 30 * 1000 ** 2), which just fits in memory, your system would also crash after a pickle.dump. That is, I had this behavior on two of my machine both running a Ubuntu setup though. Nevertheless,

[issue25465] Pickle uses O(n) memory overhead

Eric V. Smith added the comment: In what way does the OS crash? Are there any kernel messages? Or is this the python executable crashing? Again, if so, what messages are printed? In any event, if this really is an OS crash, then it's a Linux bug and should be reported to them. --

[issue25465] Pickle uses O(n) memory overhead

Herbert added the comment: That sound reasonable regarding why O(n), but it does not explain why linux crashes (I've seen this on two ubuntu systems)if pickle runs out of memory. -- ___ Python tracker

[issue25465] Pickle uses O(n) memory overhead

Serhiy Storchaka added the comment: That is because a pickler keeps track of all pickled objects. This is needed to preserve identity and support recursive objects. You can disable memoizing by setting the "fast" attribute of the Pickler object. def fastdump(obj, file): p =

[issue25465] Pickle uses O(n) memory overhead

Changes by Herbert : -- type: -> performance versions: +Python 3.4 ___ Python tracker ___

[issue25465] Pickle uses O(n) memory overhead

New submission from Herbert: I very often want to use pickle to store huge objects, such that I do not need to recalculate them again. However, I noticed that pickle uses O(n) (for n the object size in memory) amount of memory. That is, using python 3: data = {'%06d' % i: i for i in