I'm working on some performance patches for cPickle, and one of the bigger wins so far has been replacing the Pickler's memo dict with a custom hashtable (and hence removing memo's getters and setters). In looking over this, Jeffrey Yasskin commented that this would break anyone who was accessing the memo attribute.
I've found a few examples of code using the memo attribute ([1], [2], [3]), and there are probably more out there, but the memo attribute doesn't look like part of the API to me. It's only documented in http://docs.python.org/library/pickle.html as "you used to need this before Python 2.3, but don't anymore". However: I don't believe you should ever need this attribute. The usages of memo I've seen break down into two camps: clearing the memo, and wanting to explicitly populate the memo with predefined values. Clearing the memo is recommended as part of reusing Pickler objects, but I can't fathom when you would want to reuse a Pickler *without* clearing the memo. Reusing the Pickler without clearing the memo will produce pickles that are, as best I can see, invalid -- at least, pickletools.dis() rejects this, which is the closest thing we have to a validator. Explicitly setting memo values has the same problem: an easy, very brittle way to produce invalid data. So my questions are these: 1) Should Pickler/Unpickler objects automatically clear their memos when dumping/loading? 2) Is memo an intentionally exposed, supported part of the Pickler/Unpickler API, despite the lack of documentation and tests? Thanks, Collin [1] - http://google.com/codesearch/p?hl=en#Qx8E-7HUBTk/trunk/google/appengine/api/memcache/__init__.py&q=lang:py%20%5C.memo [2] - http://google.com/codesearch/p?hl=en#M-DDI-lCOgE/lib/python2.4/site-packages/cvs2svn_lib/primed_pickle.py&q=lang:py%20%5C.memo [3] - http://google.com/codesearch/p?hl=en#l_w_cA4dKMY/AtlasAnalysis/2.0.3-LST-1/PhysicsAnalysis/PyAnalysis/PyAnalysisUtils/python/root_pickle.py&q=lang:py%20pick.*%5C.memo%5Cb _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
