Re: [ZODB-Dev] ZODB alternate serialization format patch
On Jan 15, 2009, at 3:21 AM, Shane Hathaway wrote: > Jim and ZODB Developers, > > I have just created a patch for ZODB that makes the object > serialization format pluggable, meaning that it should now be > possible to write Python code that causes ZODB to store in formats > other than Python pickles. > > I have created two versions of the patch, one for ZODB 3.8 and one > for the ZODB trunk: > > http://svn.zope.org/Sandbox/shane/serialization-format-ZODB-3-8.patch?view=markup > http://svn.zope.org/Sandbox/shane/serialization-format-ZODB-trunk.patch?view=markup > > In addition to making the data format pluggable, the patch puts most > of ZODB's dependencies on the cPickle module in one place, so now if > we decide to improve our usage of the cPickle module, we can make > that change in one place rather than several. Revision 92821, for > example, probably would have required a change to only one module. > > Revision 92821: http://svn.zope.org/ZODB/trunk?rev=92821&view=rev > > How would you feel about including this in ZODB 3.9? I'll look at it and let you know. > Note that this is completely distinct from the shane-poll- > invalidations branch that RelStorage needs, although the two patches > are intended to be complementary. OK. I still need to review that branch. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB alternate serialization format patch
Hanno Schlichting wrote: > Shane Hathaway wrote: >> I have just created a patch for ZODB that makes the object serialization >> format pluggable, meaning that it should now be possible to write Python >> code that causes ZODB to store in formats other than Python pickles. > > Cool :) > >> In addition to making the data format pluggable, the patch puts most of >> ZODB's dependencies on the cPickle module in one place, so now if we >> decide to improve our usage of the cPickle module, we can make that >> change in one place rather than several. > > I'm wondering if this would be a good opportunity to include a version > marker in addition to the protocol format to the API's? Maybe. Starting with pickle protocol version 2, pickles do start with a version header. > The ZODB still uses version one of the pickle protocol throughout and so > far it has been complicated and cumbersome to change this in any way. Is it cumbersome? I tried changing to pickle protocol v2 while making this patch. Only 4 of the 3000+ ZODB tests failed. One of them failed because we have some pickle introspection code that does not yet understand protocol 2; that should be easy to fix. Another failed because it seems to depend on the exact length of a generated pickle, which is a bit silly. I didn't look at the rest of the test failures because I assume they are all similarly superficial. > While the documentation of Protocol Buffers mention that they try to be > stable and avoid incompatible versions, I don't trust any standard to be > so generic that it can avoid incompatible changes over a period of many > years. The patch I made might cover this. After this patch, ZODB will look for the serializer format name in curly braces at the beginning of each serialized object. That format name could include a version number if necessary. BTW, I've learned that Google's protocol buffers lack the concept of a mapping, and I think that might be a major problem for my current project. We might switch to a different language agnostic format like Thrift or JSON. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB alternate serialization format patch
Shane Hathaway wrote: > I have just created a patch for ZODB that makes the object serialization > format pluggable, meaning that it should now be possible to write Python > code that causes ZODB to store in formats other than Python pickles. Cool :) > In addition to making the data format pluggable, the patch puts most of > ZODB's dependencies on the cPickle module in one place, so now if we > decide to improve our usage of the cPickle module, we can make that > change in one place rather than several. I'm wondering if this would be a good opportunity to include a version marker in addition to the protocol format to the API's? The ZODB still uses version one of the pickle protocol throughout and so far it has been complicated and cumbersome to change this in any way. While the documentation of Protocol Buffers mention that they try to be stable and avoid incompatible versions, I don't trust any standard to be so generic that it can avoid incompatible changes over a period of many years. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB alternate serialization format patch
Jim and ZODB Developers, I have just created a patch for ZODB that makes the object serialization format pluggable, meaning that it should now be possible to write Python code that causes ZODB to store in formats other than Python pickles. I have created two versions of the patch, one for ZODB 3.8 and one for the ZODB trunk: http://svn.zope.org/Sandbox/shane/serialization-format-ZODB-3-8.patch?view=markup http://svn.zope.org/Sandbox/shane/serialization-format-ZODB-trunk.patch?view=markup In addition to making the data format pluggable, the patch puts most of ZODB's dependencies on the cPickle module in one place, so now if we decide to improve our usage of the cPickle module, we can make that change in one place rather than several. Revision 92821, for example, probably would have required a change to only one module. Revision 92821: http://svn.zope.org/ZODB/trunk?rev=92821&view=rev How would you feel about including this in ZODB 3.9? Note that this is completely distinct from the shane-poll-invalidations branch that RelStorage needs, although the two patches are intended to be complementary. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
