On Thu, Jun 14, 2012 at 5:34 PM, Ralf Hauenschild
> Unfortunately, my aim using ZODB was, to store dictionaries of a size of
> ~3GB memory size to the hard drive, after the data has been read from there.
> Of course, this makes only sense, if retrieving the dictionaries from ZODB
> back to the RAM goes faster than parsing the original files via readline()
> This could not be accomplished:
> The transaction.commit() alone took over an our for a dictionary, which was
> initially parsed from a file in 5 Minutes!
> So i guess, using ZODB for large files is not recommended. But in the case
> of small files, my RAM is big enough anyway, so unfortunately ZODB should
> have no use for me at the moment, or should it?
Do not store large items as persistent objects, no. Your options:
* Store as a ZODB blob, if this is one chunk of data.
* Store as a tree of persistent objects if parts need changing over
time. Future commits would be a lot smaller that way. I'd parse out
the original large dictionary in a separate process, perhaps chunk the
commits (so build up the structure over several commits).
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org