[issue25465] Pickle uses O(n) memory overhead

[Stefan Krah]

Stefan Krah added the comment:

It's a Linux issue. Disable overcommitting of memory (at your own
peril) or set user limits (for example with djb's softlimit), then
the process will be killed instead of freezing the machine.

--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is a workaround for memory consumption, and Linux freezing is not Python 
issue.

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Herbert]

Herbert added the comment:

It may be fair to note that I have no swap installed on one of the machines, 
just 16GB of RAM, on which the 'crash' happens. Hence I'm not sure how this 
affects paging, I would think there is no paging if there is no swap.

I can verify that the machine is 'stuck' for more than just several minutes (at 
least 30 minutes), nevertheless cannot confirm if this is due to the desktop 
environment or actually the kernel. I would agree to verify this first when I 
have access to the specific machines again.

Thank you for your input so far!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Lukas Lueg]

Lukas Lueg added the comment:

I very strongly doubt that it actually crashes your kernel - it basically 
can't. Your desktop becomes unresponsive for up to several minutes as the 
kernel has paged out about every single bit of memory to disk, raising access 
times by several orders of magnitude. Disable your swap and try again, it will 
just die.

--
nosy: +ebfe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Martin Panter]

Martin Panter added the comment:

Perhaps by OS crash you mean either the Linux out-of-memory (OOM) killer, that 
takes a hueristic stab at killing the right process, or Linux running almost 
out of memory, and everything grinding to a halt presumably because each task 
switch needs to re-read its program off the hard disk.

If either is the case, I understand this is part of Linux’s design, called 
“memory overcommit” or something. It is possible to disable it, though I 
haven’t tried myself, and many programs (probably including Python) are 
apparently not compatible.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Herbert]

Herbert added the comment:

Hi Eric,

I would assume that for the right range-parameter (in my case 30 * 1000 ** 2), 
which just fits in memory, your system would also crash after a pickle.dump. 
That is, I had this behavior on two of my machine both running a Ubuntu setup 
though.

Nevertheless, if you give me some time I'm happy to check my dmesg and any log 
you wish. I find it strange that sometimes I get a MemoryError when I run out 
of memory (in particular when using numpy), and sometimes the system crashes 
(in particular when using other python-stuff). Therefore I don't think this is 
pickle-specific, or even if this is a bug instead of a 'feature'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Eric V. Smith]

Eric V. Smith added the comment:

In what way does the OS crash? Are there any kernel messages? Or is this the 
python executable crashing? Again, if so, what messages are printed?

In any event, if this really is an OS crash, then it's a Linux bug and should 
be reported to them.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Herbert]

Herbert added the comment:

That sound reasonable regarding why O(n), but it does not explain why linux 
crashes (I've seen this on two ubuntu systems)if pickle runs out of memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

That is because a pickler keeps track of all pickled objects. This is needed to 
preserve identity and support recursive objects.

You can disable memoizing by setting the "fast" attribute of the Pickler object.

def fastdump(obj, file):
p = pickle.Pickler(file)
p.fast = True
p.dump(obj)

But you can't pickle recursive objects in the "fast" mode.

--
nosy: +alexandre.vassalotti, pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Herbert]

Changes by Herbert :


--
type:  -> performance
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25465] Pickle uses O(n) memory overhead

[Herbert]

New submission from Herbert:

I very often want to use pickle to store huge objects, such that I do not need 
to recalculate them again.

However, I noticed that pickle uses O(n) (for n the object size in memory) 
amount of memory. That is, using python 3:

data = {'%06d' % i: i for i in range(30 * 1000 ** 2)}
# data consumes a lot of my 8GB ram
import pickle
with open('dict-database.p3', 'wb') as f: pickle.dump(data, f)
# I have to kill the process, in order to not overflow in memory. If I 
don't, the OS crashes. IMHO the OS should never crash due to python.

I don't think pickle should require a O(n) memory overhead.

--
messages: 253371
nosy: prinsherbert
priority: normal
severity: normal
status: open
title: Pickle uses O(n) memory overhead

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com