New submission from Herbert:
I very often want to use pickle to store huge objects, such that I do not need
to recalculate them again.
However, I noticed that pickle uses O(n) (for n the object size in memory)
amount of memory. That is, using python 3:
data = {'%06d' % i: i for i in range(30 * 1000 ** 2)}
# data consumes a lot of my 8GB ram
import pickle
with open('dict-database.p3', 'wb') as f: pickle.dump(data, f)
# I have to kill the process, in order to not overflow in memory. If I
don't, the OS crashes. IMHO the OS should never crash due to python.
I don't think pickle should require a O(n) memory overhead.
----------
messages: 253371
nosy: prinsherbert
priority: normal
severity: normal
status: open
title: Pickle uses O(n) memory overhead
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25465>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
