Eric Jonas [EMAIL PROTECTED] writes:
Try gc.disable() before loading the pickle, and gc.enable() after.
Is cPickle's behavior known to be O(n^2)?
No, but the garbage collector's sometimes is.
Wow, that totally fixed it -- we went from 1200 seconds to 60
seconds.
60 seconds is still a
Hello,
I've done some benchmarking while attempting to serialize my (large)
graph data structure with cPickle; I'm seeing superlinear performance
(plotting it seems to suggest n^2 where n is the number of nodes of my
graph), in the duration of the pickle.dump calls and I can't quite
figure out
Eric Jonas [EMAIL PROTECTED] writes:
I've done some benchmarking while attempting to serialize my (large)
graph data structure with cPickle; I'm seeing superlinear performance
(plotting it seems to suggest n^2 where n is the number of nodes of my
graph), in the duration of the pickle.dump
If you are getting to the point where your data is large enough to
really care about the speed of cPickle, then maybe its time you moved
past pickles for your storage format? 2.5 includes sqlite, so you
could persist them in a nice, indexed table or something. Just a
suggestion.
On Jun
On Thu, 2008-06-12 at 20:57 +0200, Hrvoje Niksic wrote:
Eric Jonas [EMAIL PROTECTED] writes:
I've done some benchmarking while attempting to serialize my (large)
graph data structure with cPickle; I'm seeing superlinear performance
(plotting it seems to suggest n^2 where n is the number
