On Aug 7, 5:26 pm, GZ <zyzhu2...@gmail.com> wrote: > I am wondering if there is a module that can persist a stream of > objects without having to load everything into memory. (For this > reason, I think Pickle is out, too, because it needs everything to be > in memory.)
>From the pickle docs it looks like you could do something like: try: import cPickle as pickle except ImportError import pickle file_obj = open('whatever', 'wb') p = pickle.Pickler(file_obj) for x in stream_of_objects: p.dump(x) p.memo.clear() del p file_obj.close() then later file_obj = open('whatever', 'rb') p = pickle.Unpickler(file_obj) while True: try: x = p.load() do_something_with(x) except EOFError: break Your loading loop could be wrapped in a generator function, so only one object should be held in memory at once. -- http://mail.python.org/mailman/listinfo/python-list