Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Friday, May 10, 2013 11:01:33 PM Tres Seaver wrote: > > Tres, are we ready to commit to a zope.security 4.0.0 as well? > > AFAIK, we could cut it from the 'master' at any time. I don't know of > any issues I don't see any open launchpad issues which should block a > release. Cool, we can do that as part of the package finalizations of all the ZTK packages. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/10/2013 07:55 PM, Stephan Richter wrote: > On Friday, May 10, 2013 05:04:31 PM Tres Seaver wrote: >> I pushed out a ZODB 4.0.0b1 release after the merge. If the >> buildbots stay green over the weekend, I think we can release a >> 4.0.0 final early next week. > > Awesome thanks. After the 4.0.0 final release, I will set aside some > time for the CH devs to release final versions of all the ZTK > packages. > > Tres, are we ready to commit to a zope.security 4.0.0 as well? AFAIK, we could cut it from the 'master' at any time. I don't know of any issues I don't see any open launchpad issues which should block a release. tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNtIUACgkQ+gerLs4ltQ6mkACffp0He4ybnf0dkTJwLCiJxtjy b7EAn0S6iQX6sgroQ66VFfFMXeyQOZob =eY19 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Friday, May 10, 2013 05:04:31 PM Tres Seaver wrote: > I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots > stay green over the weekend, I think we can release a 4.0.0 final early > next week. Awesome thanks. After the 4.0.0 final release, I will set aside some time for the CH devs to release final versions of all the ZTK packages. Tres, are we ready to commit to a zope.security 4.0.0 as well? Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Fri, May 10, 2013 at 5:04 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 05/08/2013 12:34 PM, Tres Seaver wrote: >> On 04/29/2013 08:37 PM, Stephan Richter wrote: >>> Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready >>> for Py3 with noload() support. I totally agree that we do not need >>> to solve any of the transition work now. >> >>> So for ZODB Py3 support we need to: >> >>> 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just >>> contain the cPickle code that is Py3 compatible. >> >>> I do not care whether this happens for ZODB 4.0 or 4.1 as long as I >>> get some commitment that 4.1 >> >> Chris and I chatted with Jim about this over beers last Friday. I >> explained that the current 'py3; branch does not require the >> 'zodbpickle everywhere' stuff (the Python2 side doesn't use >> 'zodbpickle'). Jim then agreed that we could merge that branch before >> releasing 4.0. We will need to add some caveats to the docs / >> changelog (Python3 support is only for new applications, no forward- / >> backward-compatibility for data, etc.) >> >> Given that ZODB won't import or use 'zodbpickle' under Python2, I >> don't think we need to remove the current Python2 support (as released >> in 0.4.1): the Python3 version (with noload()) has been there all >> along. > > > I have merged the 'py3' branch to 'master': > > - - All tests pass under all four platforms using buildout. > > - - All unit tests pass on all four platforms using 'setup.py test'. > > I added the following note to the changelog: > >ZODB 4.0.x is supported on Python 3.x for *new* applications only. >Due to changes in the standard library's pickle support, the Python3 >support does **not** provide forward- or backward-compatibility >at the data level with Python2. A future version of ZODB may add >such support. > >Applications which need migrate data from Python2 to Python3 should >plan to script this migration using separte databases, e.g. via a >"dump-and-reload" approach, or by providing explicit fix-ups of the >pickled values as transactions are copied between storages. > > I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots > stay green over the weekend, I think we can release a 4.0.0 final early > next week. Great, thanks! Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/08/2013 12:34 PM, Tres Seaver wrote: > On 04/29/2013 08:37 PM, Stephan Richter wrote: >> Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready >> for Py3 with noload() support. I totally agree that we do not need >> to solve any of the transition work now. > >> So for ZODB Py3 support we need to: > >> 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just >> contain the cPickle code that is Py3 compatible. > >> I do not care whether this happens for ZODB 4.0 or 4.1 as long as I >> get some commitment that 4.1 > > Chris and I chatted with Jim about this over beers last Friday. I > explained that the current 'py3; branch does not require the > 'zodbpickle everywhere' stuff (the Python2 side doesn't use > 'zodbpickle'). Jim then agreed that we could merge that branch before > releasing 4.0. We will need to add some caveats to the docs / > changelog (Python3 support is only for new applications, no forward- / > backward-compatibility for data, etc.) > > Given that ZODB won't import or use 'zodbpickle' under Python2, I > don't think we need to remove the current Python2 support (as released > in 0.4.1): the Python3 version (with noload()) has been there all > along. I have merged the 'py3' branch to 'master': - - All tests pass under all four platforms using buildout. - - All unit tests pass on all four platforms using 'setup.py test'. I added the following note to the changelog: ZODB 4.0.x is supported on Python 3.x for *new* applications only. Due to changes in the standard library's pickle support, the Python3 support does **not** provide forward- or backward-compatibility at the data level with Python2. A future version of ZODB may add such support. Applications which need migrate data from Python2 to Python3 should plan to script this migration using separte databases, e.g. via a "dump-and-reload" approach, or by providing explicit fix-ups of the pickled values as transactions are copied between storages. I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots stay green over the weekend, I think we can release a 4.0.0 final early next week. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNYN8ACgkQ+gerLs4ltQ5GPACaA3EtwgZOUsvWmsoGOTWi0bpw n5kAoNNUVfsofX/eKxFZKd3F/pLQvvDG =7oxn -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Wednesday, May 08, 2013 12:34:00 PM Tres Seaver wrote: > > I do not care whether this happens for ZODB 4.0 or 4.1 as long as I > > get some commitment that 4.1 > > Chris and I chatted with Jim about this over beers last Friday. I > explained that the current 'py3; branch does not require the 'zodbpickle > everywhere' stuff (the Python2 side doesn't use 'zodbpickle'). Jim then > agreed that we could merge that branch before releasing 4.0. We will > need to add some caveats to the docs / changelog (Python3 support is only > for new applications, no forward- / backward-compatibility for data, etc.) > > Given that ZODB won't import or use 'zodbpickle' under Python2, I don't > think we need to remove the current Python2 support (as released in > 0.4.1): the Python3 version (with noload()) has been there all along. Sounds great, thanks! Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 08:37 PM, Stephan Richter wrote: > Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready > for Py3 with noload() support. I totally agree that we do not need to > solve any of the transition work now. > > So for ZODB Py3 support we need to: > > 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just > contain the cPickle code that is Py3 compatible. > > I do not care whether this happens for ZODB 4.0 or 4.1 as long as I > get some commitment that 4.1 Chris and I chatted with Jim about this over beers last Friday. I explained that the current 'py3; branch does not require the 'zodbpickle everywhere' stuff (the Python2 side doesn't use 'zodbpickle'). Jim then agreed that we could merge that branch before releasing 4.0. We will need to add some caveats to the docs / changelog (Python3 support is only for new applications, no forward- / backward-compatibility for data, etc.) Given that ZODB won't import or use 'zodbpickle' under Python2, I don't think we need to remove the current Python2 support (as released in 0.4.1): the Python3 version (with noload()) has been there all along. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGKfngACgkQ+gerLs4ltQ500gCfQK4HSzemxaYkcPAyleNdkagq MAwAn3wYoCo4BItBHAve4o+lhrzRTBrt =BbCU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Monday, April 29, 2013 01:15:29 PM Tres Seaver wrote: > FWIW, we have reports that some brave souls have successfully built Py3k > apps using the 'py3' branch. Yep, see here: https://github.com/CipherHealth/cipher.uibuilderdemo This is not an app in production, but shows off some features of an app in production. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Monday, April 29, 2013 01:50:22 PM Jim Fulton wrote: > Whether 4.0 supports Python 3 or not, let's quickly get to the point where > tests are run and pass on both Python 2 and 3. Once we get to that point, > we won't accept pull requests that break Python 3 (or 2, of course). > But let's get to the point soon where we can make Python 2 releases > with confidence. Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready for Py3 with noload() support. I totally agree that we do not need to solve any of the transition work now. So for ZODB Py3 support we need to: 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just contain the cPickle code that is Py3 compatible. I do not care whether this happens for ZODB 4.0 or 4.1 as long as I get some commitment that 4.1 with Py3 support is happening quickly (i.e. not in the order of many months). Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Mon, Apr 29, 2013 at 1:15 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 12:44 PM, Jim Fulton wrote: >> Yes. There are lots of features I'd like to add to ZODB. I tend to >> work on them when I have time (infrequently) or where we have a >> driving need at ZC. Long ZODB release cycles provide a lot of stop >> energy. > > We are already developing this way (the 'py3' branch has not been > merged). However, if you do a lot of Python2-only feature work and merge > to master, you will likely push back to horizon for merging that branch: > we will have to port any work done to it. I'd be happy to say that anything pushed to master has to pass tests on Python 3. I have no interest in delaying Python 3 work. > > Using the "4.0" label to signal "big changes ahead, evaluate carefully > before upgrading" was the primary reason I had been pushing to get the > Py3k stuff landed Apparently. :) But, IIRC, you never discussed this with me. When I announced 4.0, the big change was splitting off ZEO, persistent, and later BTrees. In fact, as you may remember, I suggested splitting BTrees off in 5, because I didn't want to to delay 4. > (the low-risk thing would have been more naturally > labeled "3.11"). Except I explicitly said that 4.0 was supposed to be a low risk release. That's why 3.11 was just a meta-release to aid people in the transition to 4. When I saw all your activity on porting to Python 3, I stepped back to give you room. But now, several months have gone by and we're more or less where we were in November wrt 4.0. I greatly appreciate and support you guys have done on Python 3 porting. I don't mean to criticize the work you've done. If anyone deserves criticism, it's me for not staying on top of this. We need to get to a point where we can release frequently, with confidence. That doesn't mean we will; it depends on people's time to contribute. But we need to be able and we need to plan our activities so we can release frequently. Whether 4.0 supports Python 3 or not, let's quickly get to the point where tests are run and pass on both Python 2 and 3. Once we get to that point, we won't accept pull requests that break Python 3 (or 2, of course). But let's get to the point soon where we can make Python 2 releases with confidence. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 12:44 PM, Jim Fulton wrote: > Yes. There are lots of features I'd like to add to ZODB. I tend to > work on them when I have time (infrequently) or where we have a > driving need at ZC. Long ZODB release cycles provide a lot of stop > energy. We are already developing this way (the 'py3' branch has not been merged). However, if you do a lot of Python2-only feature work and merge to master, you will likely push back to horizon for merging that branch: we will have to port any work done to it. Using the "4.0" label to signal "big changes ahead, evaluate carefully before upgrading" was the primary reason I had been pushing to get the Py3k stuff landed (the low-risk thing would have been more naturally labeled "3.11"). FWIW, we have reports that some brave souls have successfully built Py3k apps using the 'py3' branch. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+qrEACgkQ+gerLs4ltQ5OpgCg1FZ8xB9EijUQOQqzhm2RAZ4V +z4An29kZZYpoFlFeu0QMpLbqGRfIscV =5EKv -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Mon, Apr 29, 2013 at 12:25 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 11:00 AM, Jim Fulton wrote: >> Let's keep master stable. Maybe someone will want to add features >> before the Python 3 support is stable. I don't want to hold 4.1 >> hostage either. > > Given that the only folks (besides maybe you) invested in ZODB > development want Py3k support ASAP. I don't see that. Do you have > features in mind that you would imagine releasing before we land Py3k > support? Yes. There are lots of features I'd like to add to ZODB. I tend to work on them when I have time (infrequently) or where we have a driving need at ZC. Long ZODB release cycles provide a lot of stop energy. The only way to get away from long release cycles is to have a stable master that's releasable at any time. OTOH, ZODB is pretty critical software, so we have to be very confident in what we merge to master. > >> I suggest breaking the Python 3 work into increments that can each be >> introduced without sacrificing stability. >> >> The first increment could provide Python 3 support without any >> conversion or compatibility support. This is something you could >> probably achieve pretty quickly and would allow you meet your >> immediate goals. > > We are already there, AFAIK, on the 'py3' branch: the blocker is just > getting out a release. All I ask is a stable releasable master. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 11:00 AM, Jim Fulton wrote: > Let's keep master stable. Maybe someone will want to add features > before the Python 3 support is stable. I don't want to hold 4.1 > hostage either. Given that the only folks (besides maybe you) invested in ZODB development want Py3k support ASAP. I don't see that. Do you have features in mind that you would imagine releasing before we land Py3k support? > I suggest breaking the Python 3 work into increments that can each be > introduced without sacrificing stability. > > The first increment could provide Python 3 support without any > conversion or compatibility support. This is something you could > probably achieve pretty quickly and would allow you meet your > immediate goals. We are already there, AFAIK, on the 'py3' branch: the blocker is just getting out a release. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+nugACgkQ+gerLs4ltQ4lmACgmfAMS0tqV86+v0ItIlbMkhtK i6gAoLPO4aehpHIrZokPG8c4hRIAOsnE =6f/j -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Mon, Apr 29, 2013 at 10:54 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 10:51 AM, Jim Fulton wrote: > >> Right. As I suggested, let's get to a point where we can get a stable >> ZODB 4.0 release for Python 2. As soon as we get that, let's get a >> ZODB 4.0.x or 4.1 release that works on Python 3, presumably via >> zodbpickle. > > As I proposed earlier this morning, I can only reply to one email at a time. :) > we can make a non-Py3k, > non-zodbpickle 4.0b1 release today from the master branch, and a 4.0 > final in a week. > > Once we get that release out, we can then merge the 'py3' branch, > including adding the requirement for 'zodbpickle' under both Python2 and > Py3k, and aim for a much expedited 4.1 release which supports Py3k. I'd rather keep this would on a branch until it's known to be stable. I suggest instead focusing on getting new Python 3 applications working without affecting Python 2 apps. IOW, only use zodbpickle for Python 3 *initially*. I want to be able to release from master at almost any time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Mon, Apr 29, 2013 at 10:20 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 09:48 AM, Jim Fulton wrote: >> On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter >> wrote: >>> On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: Can ZODB 4 be used now without zodbpickle? >>> >>> No, unfortunately for Py2 we need the custom cPickle and for Py3 >>> `noload()` support (as Tres mentioned). >> >> This is a problem. >> >> The only change in ZODB 4.0 was supposed to be the breakup. >> >> This was supposed to be a low-risk release. The separation into >> multiple packages was supposed to increase agility, but now it appears >> we're stuck. > > The only reason we had delayed the 4.0 release (in my mind, anyway) was > that it was a good way to signal the Py3k compatibliity changes. That was a bad idea. Unless you want to reinforce the fact that Python 3 is an agility killer. .5 ;) There's release meta data to signal Python 3 compatibility. > I'm not > wedded to calling the Py3k-compatible release "4.0". Cool. >> I'd like there to a stable 4.0 release **soon** that doesn't use >> zodbpickle for Python 2. >> >> For now, I suggest we focus on stability and the ability to make >> progress on non-Python-3-related work. >> >> After that is achieved, I suggest we get to the point where people >> can create new databases and use them with Python 3. We need to do >> this without hindering the ability to make new stable releases. > > The trunk of the 'ZODB' package does not have any of the Py3k / > zodbpickle changes yet. We could make a ZODB 4.0b1 release from the > trunk today +1. > and create a '4.0' stable branch prior to any merge of the > 'py3' work. Let's keep master stable. Maybe someone will want to add features before the Python 3 support is stable. I don't want to hold 4.1 hostage either. I suggest breaking the Python 3 work into increments that can each be introduced without sacrificing stability. The first increment could provide Python 3 support without any conversion or compatibility support. This is something you could probably achieve pretty quickly and would allow you meet your immediate goals. >> As far as the grander vision for Python2/3 transition and >> interoperability, we need to make progress incrementally and not >> sacrifice stability of the master branch. >> >> I made the 3.11 release fully expecting a stable 4.0 release soon. > > That was of the 'ZODB3' meta-package, right? Yes. It was predicated on a stable 4.0 release that had very little in it beyond the split into separate packages. It was intended to help people start transitioning from ZODB3 to ZODB, but that can't happen until ZODB is stable. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 10:51 AM, Jim Fulton wrote: > Right. As I suggested, let's get to a point where we can get a stable > ZODB 4.0 release for Python 2. As soon as we get that, let's get a > ZODB 4.0.x or 4.1 release that works on Python 3, presumably via > zodbpickle. As I proposed earlier this morning, we can make a non-Py3k, non-zodbpickle 4.0b1 release today from the master branch, and a 4.0 final in a week. Once we get that release out, we can then merge the 'py3' branch, including adding the requirement for 'zodbpickle' under both Python2 and Py3k, and aim for a much expedited 4.1 release which supports Py3k. At this point, I would favor putting the database conversion / straddling code into a new package (not 'zodbpickle'). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+iZ4ACgkQ+gerLs4ltQ7SgACfW0UMVtp1MJGplWDEBfx3CM9V pesAoNqece3ZQ/EVoFad6G2+B3vXkCz8 =ig9W -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Mon, Apr 29, 2013 at 10:24 AM, Stephan Richter wrote: > On Monday, April 29, 2013 09:48:05 AM Jim Fulton wrote: >> I'd like there to a stable 4.0 release **soon** >> that doesn't use zodbpickle for Python 2. > > I would like to agree. But on the other hand, the ZODB release cycles are very > long and the prospect of waiting another 6-12 months before any Python 3 > support lands, is really scary because it prohibits me to even write a new > project in Python 3. As stated here: https://mail.zope.org/pipermail/zodb-dev/2012-October/014770.html I was hoping that the breakup of the ZODB packages would allow us to increase the tempo of releases. But increasing tempo is only possible of master is stable. > (CH has just invested about 6 man-months into the porting > effort and without ZODB we are basically stuck. But we do not need a > transition > plan, since we can recreate our ZODBs from configuration files.) > > Could we compromise and support Python 3 in ZODB 4.0 without necessarily solve > all the migration strategy issues? I suggested that in the part fo my email that you snipped. > In fact, by using zodbpickle, zodbpickle > can have a separate, faster release cycle experimenting with some transition > strategies. Maybe one way to install ZODB 4.0 would be to not use zodbpickle > and use cPickle instead. We already have all that stuff separated into a > _compat module, so that should not be too hard. Right. As I suggested, let's get to a point where we can get a stable ZODB 4.0 release for Python 2. As soon as we get that, let's get a ZODB 4.0.x or 4.1 release that works on Python 3, presumably via zodbpickle. While we want to make progress on Python 3, we can't hold ZODB hostage to the Python 3 porting effort. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 09:48 AM, Jim Fulton wrote: > On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter > wrote: >> On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: >>> Can ZODB 4 be used now without zodbpickle? >> >> No, unfortunately for Py2 we need the custom cPickle and for Py3 >> `noload()` support (as Tres mentioned). > > This is a problem. > > The only change in ZODB 4.0 was supposed to be the breakup. > > This was supposed to be a low-risk release. The separation into > multiple packages was supposed to increase agility, but now it appears > we're stuck. The only reason we had delayed the 4.0 release (in my mind, anyway) was that it was a good way to signal the Py3k compatibliity changes. I'm not wedded to calling the Py3k-compatible release "4.0". > I'd like there to a stable 4.0 release **soon** that doesn't use > zodbpickle for Python 2. > > For now, I suggest we focus on stability and the ability to make > progress on non-Python-3-related work. > > After that is achieved, I suggest we get to the point where people > can create new databases and use them with Python 3. We need to do > this without hindering the ability to make new stable releases. The trunk of the 'ZODB' package does not have any of the Py3k / zodbpickle changes yet. We could make a ZODB 4.0b1 release from the trunk today and create a '4.0' stable branch prior to any merge of the 'py3' work. > As far as the grander vision for Python2/3 transition and > interoperability, we need to make progress incrementally and not > sacrifice stability of the master branch. > > I made the 3.11 release fully expecting a stable 4.0 release soon. That was of the 'ZODB3' meta-package, right? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+gboACgkQ+gerLs4ltQ4+7wCg222VrN5b0jkRrSJKVBL1VEBr 5lgAoINrzLbTus6ycBXcVGovxWIPBQ5t =XLcz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Monday, April 29, 2013 09:48:05 AM Jim Fulton wrote: > I'd like there to a stable 4.0 release **soon** > that doesn't use zodbpickle for Python 2. I would like to agree. But on the other hand, the ZODB release cycles are very long and the prospect of waiting another 6-12 months before any Python 3 support lands, is really scary because it prohibits me to even write a new project in Python 3. (CH has just invested about 6 man-months into the porting effort and without ZODB we are basically stuck. But we do not need a transition plan, since we can recreate our ZODBs from configuration files.) Could we compromise and support Python 3 in ZODB 4.0 without necessarily solve all the migration strategy issues? In fact, by using zodbpickle, zodbpickle can have a separate, faster release cycle experimenting with some transition strategies. Maybe one way to install ZODB 4.0 would be to not use zodbpickle and use cPickle instead. We already have all that stuff separated into a _compat module, so that should not be too hard. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter wrote: > On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: >> Can ZODB 4 be used now without zodbpickle? > > No, unfortunately for Py2 we need the custom cPickle and for Py3 `noload()` > support (as Tres mentioned). This is a problem. The only change in ZODB 4.0 was supposed to be the breakup. This was supposed to be a low-risk release. The separation into multiple packages was supposed to increase agility, but now it appears we're stuck. I'd like there to a stable 4.0 release **soon** that doesn't use zodbpickle for Python 2. For now, I suggest we focus on stability and the ability to make progress on non-Python-3-related work. After that is achieved, I suggest we get to the point where people can create new databases and use them with Python 3. We need to do this without hindering the ability to make new stable releases. As far as the grander vision for Python2/3 transition and interoperability, we need to make progress incrementally and not sacrifice stability of the master branch. I made the 3.11 release fully expecting a stable 4.0 release soon. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: > Can ZODB 4 be used now without zodbpickle? No, unfortunately for Py2 we need the custom cPickle and for Py3 `noload()` support (as Tres mentioned). Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/28/2013 07:19 PM, Jim Fulton wrote: > I'm confused. I don't understand why we need a Python 3 pickler > change to support the new Python 2 binary type. I thought we were > going to pickle Python 2 binary objects using the standard Python 3 > protocol 3 code? We need 'noload' support on Py3k. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF9sc0ACgkQ+gerLs4ltQ40kwCfeH++IWblTJRNDyma/OZQkGX6 tYwAmQFCs5ynIDzONiBKbqC//sD0MdUk =0xq4 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Fri, Apr 26, 2013 at 8:02 PM, Stephan Richter wrote: > On Friday, April 26, 2013 05:34:15 PM Tres Seaver wrote: >> I would like to merge this branch to master early next week and make a >> release, so that we can evaluate merging the 'py3' branch of ZODB. >> >> Thoughts? Note that I have not yet addressed the portions of my proposal >> which deal with analyzing / converting existing databases, or with the >> possibly-needed wrapper storage (for on-the-fly conversion). > > Let's do it. This way people can test ZODB 4 on their existing Py2 code bases > and we at CH can test our uibuilder/demo on Python 3. This will give us at > least some confidence that we are going in the right direction. > > We might consider not even tackling DB conversions for ZODB 4.0 and delay that > to ZODB 4.1 or leave it even up to an add-on package. This way people can > experiment with different approaches and we do not have to nail the conversion > problem with ZODB 4.0. I've lost track of where we are with ZODB 4. Can ZODB 4 be used now without zodbpickle? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Wed, Apr 17, 2013 at 2:54 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/16/2013 05:13 PM, Stephan Richter wrote: >> On Tuesday, April 16, 2013 04:38:06 PM Tres Seaver wrote: >>> Comments? > > (I don't now why Stephan's e-mail didn't make it to the list). > > >> The big omission that I noticed while reading the text carefully is a >> note saying that you will never be able to use stock Py3k pickle, >> because it does not support noload(). Thus ``zodbpickle`` is needed >> for any Py3k code. (I think this is a correction to you your last >> bullet in _replace_py2_cPickle.) > > Hmm, I think you are correct. > >> That reminds me, originally we forked pickle.py from Python 3.3. >> During PyCon I think you decided to start by using cPickle from Python >> 2.7 instead. If you are starting from Py2.7 cPickle, then supporting >> Protocol 3 is not easy. > > Already done (as you note in your follow-up). > >> Given your writeup, I think you are implicitly saying to start from >> Py3.3 pickle and add the special support for Python 2 binary via the >> special new type. That sounds good to me. > > I would actually prefer to fork the Python 3.2 version: the one from 3.3 > pulls in a bunch of grotty internal-only usage. I'm confused. I don't understand why we need a Python 3 pickler change to support the new Python 2 binary type. I thought we were going to pickle Python 2 binary objects using the standard Python 3 protocol 3 code? >> BTW, what are your motivations for all the different strategies? > > I wanted to document them all, because some of the strategies suit > different cases better than others. > >> _ignore_compat is obvious. If you can easily create the ZODB from >> other data sources, then you can do a one-time switch. In fact, at >> CipherHealth we have this case, since the ZODB only contains config >> (which is loaded from text files) and session data. > > Yup. Even for large CMS systems, I would still make "dump-to-filesystem, > then reload" a requirement. Others disagree, of course (and may have > legitimate reasons). Leo Rochael Almeida has clients with databases "too > big to convert", for instance (the downtime required to do the conversion > would be prohibitive, I believe). > >> But which strategy would be useful for a large Plone site for example? >> I think we should focus on that and provide one good way to do it. > > Plone has historically preferred in-place migration to dump-reload. Maybe > jumping the Py3k curb is enough reason for them to reconsider. I'm hoping to be able to provide some help with in-place conversion in the near future. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Tue, Apr 16, 2013 at 4:38 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > After getting a bit bogged down during the PyCon US 2013 sprints, I'd > like to restart the discussion by outlining the problem as I think I > understand it now. > > Proposal for ZODB pickle compatibility > == > > Issues > - -- > > - - There exists no forward-compatible way to pickle bytes on Python2 > (Py3k pickle module "guesses", decoding any Python2 ``str`` using > ``latin1``). > > - - Some data pickled as ``str`` on Python2 truly is binary (e.g., > ``Pdata`` objects for Zope2's ``OFS.Image.File`` and > ``OFS.Image.Image`` types; crypto hases?) > > - - Some Python2 applications may have the same attribute for a given > class stored both as ``str`` and as ``unicode`` (due e.g., to bugs in > the code, literal defaults, browser quirks, changes to code over > time). > > > Scenarios > - - > > > .. _py2_forever: > > Existing Python2-only Application > + > > - - Code for the app is never(ish) going to migrate to Py3k. > > - - Using an updated / supported ZODb package **must** be possible > > - - Ideally, requires no changes to application code. > > - - Ideally, requies no database fixup / conversion. > > - - Best strategy is likely ignore_compat_. > > > .. _py3k_only: > > New, Py3k-only Application > ++ > > - - Code for the app will run only on Py3k. > > - - Running with the latest-and-greatest ZODB **must** be possible. > > - - Ideally, the code for the app will make no concessions to backward- > compatibility. > > - - Best strategy is likely ignore_compat_. > > > .. _migrate_w_convert: > > Python2 Application Migrating to Py3k > + > > - - Application code "straddles" both Pythons using "compatible subset" > dialect, but only during the migration period. > > - - During that period, code **must** be able to open the database from > both Python2 and Py3k. > > - - Ideally, application code will need to make no concessions to > backward-compatibility after migration. > > - - It is acceptable to run a conversion process which normalizes all > active records in the database prior to testing. > > - - For databases which are already "binary clean" (binary data exists > only in blobs; the application creates no new non-blob binary > attributes), the best strategy is likely ignore_compat_. I don't like the idea of supporting binary data only in blobs. > > - - For databases which are not already "binary clean" (there may be > non-blob binary attributes), the best strategy is likely to > convert_storages_, followed by replace_py2_cpickle_ (if the Python2 > client might create new non-blob binary attributes). > > - - wrap_storages_ (on the Python2 side) might be simpler than > replace_py2_cpickle_, if the sources of non-blob binary attributes are > well understood. > > > .. _straddle_w_convert: > > Python2 Application Straddling Python2 / Py3k (1) > + > > - - Application code "straddles" both Pythons using "compatible subset" > dialect. > > - - Code **must** be able to open the database from both Python2 and Py3k. > > - - It is acceptable to run a conversion process which normalizes all > active records in the database prior to testing. > > - - For databases which are already "binary clean" (binary data exists > only in blobs; the application creates no new non-blob binary > attributes), the best strategy is likely ignore_compat_. > > - - For databases which are not already "binary clean" (there may be > non-blob binary attributes), the best strategy is likely to > convert_storages_, followed by replace_py2_cpickle_ (if the Python2 > client might create new non-blob binary attributes). > > - - For cases where Python2 and Py3k clients may share the database for an > extended period, and where disruption to the Python2 clients must be > minimized, the replace_py3k_pickle_ strategy might be preferred, until > convert_storages_ becomes feasible. IMO, _replace_py2_cPickle is the best strategy in this scenario. As noted above, I think it's important to support non-blob binary data. > > > .. _straddle_no_convert: > > Python2 Application Migrating to Py3k (2) > + > > - - Application code "straddles" both Pythons using "compatible subset" > dialect. > > - - Code **must** be able to open the database from both Python2 and Py3k. > > - - It is **not** acceptable to run a conversion process which normalizes > all active records in the database prior to testing (e.g., the > database is too large to convert on existing hardware, or the downtime > required for conversion is unacceptable). > > - - Because disruption to the Python2 clients must be minimized, the best > strategy is likely replace_py3k_pickle_ until convert_storages_ > b
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Friday, April 26, 2013 05:34:15 PM Tres Seaver wrote: > I would like to merge this branch to master early next week and make a > release, so that we can evaluate merging the 'py3' branch of ZODB. > > Thoughts? Note that I have not yet addressed the portions of my proposal > which deal with analyzing / converting existing databases, or with the > possibly-needed wrapper storage (for on-the-fly conversion). Let's do it. This way people can test ZODB 4 on their existing Py2 code bases and we at CH can test our uibuilder/demo on Python 3. This will give us at least some confidence that we are going in the right direction. We might consider not even tackling DB conversions for ZODB 4.0 and delay that to ZODB 4.1 or leave it even up to an add-on package. This way people can experiment with different approaches and we do not have to nail the conversion problem with ZODB 4.0. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter" ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 04:38 PM, Tres Seaver wrote: > - ``zodbpickle`` should provide a new ``binary`` type which Python2 > applications can begin using to signal that attributes should be > unpickled in Py3k as ``bytes``. See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - ``zodbpickle`` should provide a pickler/unpickler for use by Python2 > clients who operate against converted storages (replace_py2_cpickle_). > See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - ``zodbpickle`` should provide a pickler/unpickler for use by Py3k > clients who operate against unconverted storages > (replace_py3k_pickle_). See: > https://github.com/zopefoundation/zodbpickle I have pushed a new branch, 'merge_py2_py3k', which basically holds the existing Py3k forks (from master) alongside the Pyton-2 forks (from 'py2_explicit_bytes'): https://github.com/zopefoundation/zodbpickle/tree/merge_py2_py3k After thrashing and failing in my attempt to have the Python code straddle, I instead have separate versions (based on Python 2.7 and 3.2), and a unifying 'zodbpickle.pickle' module which pulls in the standard API from the appropriate module. This solution is ugly, but it does allow for running tests cleanly on each supported version of Python. It also minimized the extent of patching for each fork (the straddled version I discarded was a franken-module by comparison). I would like to merge this branch to master early next week and make a release, so that we can evaluate merging the 'py3' branch of ZODB. Thoughts? Note that I have not yet addressed the portions of my proposal which deal with analyzing / converting existing databases, or with the possibly-needed wrapper storage (for on-the-fly conversion). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF68tcACgkQ+gerLs4ltQ4lzQCdHvsnCZ5cfZlycJ1BQw8Uct5q CGYAnilMitegoqsVOevpPlbLcuu7bur6 =qEsj -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:19 PM, Stephan Richter wrote: > Oh, I see that in your branch of ``zodbpickle`` you already added > protocol 3 support to the Py2.7 cPickle code. Does this code also run > under Py3.3? (I don't now why Stephan's e-mail didn't make it to the list). I would not expect to use the Python 2.7-derived pickler on Py3k. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu7+UACgkQ+gerLs4ltQ4EWgCgyGHjPlqWEK5+PFhJXK94jCLY edMAn2aZV5dLxV7XdTgSh83yOBrdnTL/ =/tPT -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:13 PM, Stephan Richter wrote: > On Tuesday, April 16, 2013 04:38:06 PM Tres Seaver wrote: >> Comments? (I don't now why Stephan's e-mail didn't make it to the list). > The big omission that I noticed while reading the text carefully is a > note saying that you will never be able to use stock Py3k pickle, > because it does not support noload(). Thus ``zodbpickle`` is needed > for any Py3k code. (I think this is a correction to you your last > bullet in _replace_py2_cPickle.) Hmm, I think you are correct. > That reminds me, originally we forked pickle.py from Python 3.3. > During PyCon I think you decided to start by using cPickle from Python > 2.7 instead. If you are starting from Py2.7 cPickle, then supporting > Protocol 3 is not easy. Already done (as you note in your follow-up). > Given your writeup, I think you are implicitly saying to start from > Py3.3 pickle and add the special support for Python 2 binary via the > special new type. That sounds good to me. I would actually prefer to fork the Python 3.2 version: the one from 3.3 pulls in a bunch of grotty internal-only usage. > BTW, what are your motivations for all the different strategies? I wanted to document them all, because some of the strategies suit different cases better than others. > _ignore_compat is obvious. If you can easily create the ZODB from > other data sources, then you can do a one-time switch. In fact, at > CipherHealth we have this case, since the ZODB only contains config > (which is loaded from text files) and session data. Yup. Even for large CMS systems, I would still make "dump-to-filesystem, then reload" a requirement. Others disagree, of course (and may have legitimate reasons). Leo Rochael Almeida has clients with databases "too big to convert", for instance (the downtime required to do the conversion would be prohibitive, I believe). > But which strategy would be useful for a large Plone site for example? > I think we should focus on that and provide one good way to do it. Plone has historically preferred in-place migration to dump-reload. Maybe jumping the Py3k curb is enough reason for them to reconsider. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu79sACgkQ+gerLs4ltQ4aTQCfSeb6Xiz0OOtoGJuzSKfOetMu 7IAAoMiNzaohY2lv8DMoLRmoNIw8f3P0 =fcYz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
On Tue, Apr 16, 2013 at 5:38 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > [...] > > Concrete Proposal > - - > > I believe we will need to update ``zodbpickle`` and ``ZDOB`` to allow > for any of the strategies to be applied. > > - - ``zodbpickle`` should provide the script which analyzes pickles in > a database for inconsistent ``str`` / ``unicode`` usage. See: > https://github.com/jimfulton/dbstringanalysis > > - - ``zodbpickle`` should provide the utility for registering per-class > fixups. > > - - ``zodbpickle`` should provide the script which uses that utility > do to one-time conversion of a storage (supporting convert_storages_). > > - - ``zodbpickle`` should provide a new ``binary`` type which Python2 > applications can begin using to signal that attributes should be > unpickled in Py3k as ``bytes``. See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - - ``zodbpickle`` should provide a pickler/unpickler for use by > Python2 clients who operate against converted storages > (replace_py2_cpickle_). See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - - ``zodbpickle`` should provide a pickler/unpickler for use by > Py3k clients who operate against unconverted storages > (replace_py3k_pickle_). See: > https://github.com/zopefoundation/zodbpickle > > - - ``zodbpickle`` might need to provide a wrapper storage supporting > straddle_no_convert_. > > > Comments? In addition to the above, I'd like to suggest that: - - ``zodbpickle`` should provide a "class Native(str)" type to help in straddling and live conversion. This Native type would be the result of unpickling "STRING" codes, and behave as specified in: https://github.com/zopefoundation/ZODB/wiki/Pickles#explicit-but-transitional-native-string-for-python-3 > > > Tres. > - -- > === > Tres Seaver +1 540-429-0999 tsea...@palladion.com > Palladion Software "Excellence by Design"http://palladion.com > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlFttq4ACgkQ+gerLs4ltQ5fswCeLcPj7QROXzlXazJIuK/nAAf6 > YzkAnj07aERlQhZInv+lFWvQjqJnciZ8 > =PLZq > -END PGP SIGNATURE- > > ___ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 After getting a bit bogged down during the PyCon US 2013 sprints, I'd like to restart the discussion by outlining the problem as I think I understand it now. Proposal for ZODB pickle compatibility == Issues - -- - - There exists no forward-compatible way to pickle bytes on Python2 (Py3k pickle module "guesses", decoding any Python2 ``str`` using ``latin1``). - - Some data pickled as ``str`` on Python2 truly is binary (e.g., ``Pdata`` objects for Zope2's ``OFS.Image.File`` and ``OFS.Image.Image`` types; crypto hases?) - - Some Python2 applications may have the same attribute for a given class stored both as ``str`` and as ``unicode`` (due e.g., to bugs in the code, literal defaults, browser quirks, changes to code over time). Scenarios - - .. _py2_forever: Existing Python2-only Application + - - Code for the app is never(ish) going to migrate to Py3k. - - Using an updated / supported ZODb package **must** be possible - - Ideally, requires no changes to application code. - - Ideally, requies no database fixup / conversion. - - Best strategy is likely ignore_compat_. .. _py3k_only: New, Py3k-only Application ++ - - Code for the app will run only on Py3k. - - Running with the latest-and-greatest ZODB **must** be possible. - - Ideally, the code for the app will make no concessions to backward- compatibility. - - Best strategy is likely ignore_compat_. .. _migrate_w_convert: Python2 Application Migrating to Py3k + - - Application code "straddles" both Pythons using "compatible subset" dialect, but only during the migration period. - - During that period, code **must** be able to open the database from both Python2 and Py3k. - - Ideally, application code will need to make no concessions to backward-compatibility after migration. - - It is acceptable to run a conversion process which normalizes all active records in the database prior to testing. - - For databases which are already "binary clean" (binary data exists only in blobs; the application creates no new non-blob binary attributes), the best strategy is likely ignore_compat_. - - For databases which are not already "binary clean" (there may be non-blob binary attributes), the best strategy is likely to convert_storages_, followed by replace_py2_cpickle_ (if the Python2 client might create new non-blob binary attributes). - - wrap_storages_ (on the Python2 side) might be simpler than replace_py2_cpickle_, if the sources of non-blob binary attributes are well understood. .. _straddle_w_convert: Python2 Application Straddling Python2 / Py3k (1) + - - Application code "straddles" both Pythons using "compatible subset" dialect. - - Code **must** be able to open the database from both Python2 and Py3k. - - It is acceptable to run a conversion process which normalizes all active records in the database prior to testing. - - For databases which are already "binary clean" (binary data exists only in blobs; the application creates no new non-blob binary attributes), the best strategy is likely ignore_compat_. - - For databases which are not already "binary clean" (there may be non-blob binary attributes), the best strategy is likely to convert_storages_, followed by replace_py2_cpickle_ (if the Python2 client might create new non-blob binary attributes). - - For cases where Python2 and Py3k clients may share the database for an extended period, and where disruption to the Python2 clients must be minimized, the replace_py3k_pickle_ strategy might be preferred, until convert_storages_ becomes feasible. .. _straddle_no_convert: Python2 Application Migrating to Py3k (2) + - - Application code "straddles" both Pythons using "compatible subset" dialect. - - Code **must** be able to open the database from both Python2 and Py3k. - - It is **not** acceptable to run a conversion process which normalizes all active records in the database prior to testing (e.g., the database is too large to convert on existing hardware, or the downtime required for conversion is unacceptable). - - Because disruption to the Python2 clients must be minimized, the best strategy is likely replace_py3k_pickle_ until convert_storages_ becomes feasible. - - Alternatively, wrap_storages_ might be the best strategy for the Py3k clients. Strategies - -- .. _ignore_compat: Ignore compatibility Use the stdlib pickle support in its default mode. - - No changes to the ``ZODB`` packages on Python2 or Py3k. - - Pickles created under Python2 will be readable on Py3k; however, *all* bytes data will be coerced (via ``latin1``) to unicode. - - Pickles created under Py3k will likely no
