Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 12:25 AM, Nathan Van Gheem vangh...@gmail.com wrote: ... I'll look at making a 3.8 release with just this, or maybe this and a few other fixes if they are extremely low-hanging. That would be greatly appreciated. Let me know if there is anything I can do to help... 3.8.6 is available now with this bug fix and a few others. I'll be making a 3.9.6 release later today. I'm uncomfortable with with people using 3.8 with blobs in production. I'm going to spend some time today seeing if I can add back enough version api (as opposed to support :) in 3.9 to make it work with Zope 2.10. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton j...@zope.com wrote: 3.8.6 is available now with this bug fix and a few others. Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So it's not straightforward to use this in older Plone versions. Personally I used 3.8.4 in all Plone 3 versions with blobs. We've run this combination on large data sets without any problems and I know of quite a number of other Plone companies having done the same. I'm uncomfortable with with people using 3.8 with blobs in production. I'm going to spend some time today seeing if I can add back enough version api (as opposed to support :) in 3.9 to make it work with Zope 2.10. That's a very noble goal :) Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton j...@zope.com wrote: 3.8.6 is available now with this bug fix and a few others. Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So it's not straightforward to use this in older Plone versions. AFAIK, this fix doesn't convert overflow errors to type errors. It simply raises type errors in situations where data were simply stored incorrectly before. I'm surprised that this would cause problems. Personally I used 3.8.4 in all Plone 3 versions with blobs. We've run this combination on large data sets without any problems and I know of quite a number of other Plone companies having done the same. For the most part, the work I did in ZODB 3.9 was motivated by a desire to get blobs to the point that I'd be willing to use them in production. shrug I'm uncomfortable with with people using 3.8 with blobs in production. I'm going to spend some time today seeing if I can add back enough version api (as opposed to support :) in 3.9 to make it work with Zope 2.10. That's a very noble goal :) BTW, if there a standard way to get Zope 2.10 to use a version of ZODB other than the one that ships with it? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton j...@zope.com wrote: On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting ha...@hannosch.eu wrote: Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So it's not straightforward to use this in older Plone versions. AFAIK, this fix doesn't convert overflow errors to type errors. It simply raises type errors in situations where data were simply stored incorrectly before. I'm surprised that this would cause problems. It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442r1=115279r2=115442 BTW, if there a standard way to get Zope 2.10 to use a version of ZODB other than the one that ships with it? It's all just a PYTHONPATH in the end. So if you put something on the path by the name of ZODB first, then the one in the software home inside Zope2 won't be loaded. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 11:16 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton j...@zope.com wrote: On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting ha...@hannosch.eu wrote: Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So it's not straightforward to use this in older Plone versions. AFAIK, this fix doesn't convert overflow errors to type errors. It simply raises type errors in situations where data were simply stored incorrectly before. I'm surprised that this would cause problems. It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442r1=115279r2=115442 I don't understand how the BTree changes would run afoul of this. This code should prevent the BTree from overflowing in the first place, no? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton j...@zope.com wrote: It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442r1=115279r2=115442 I don't understand how the BTree changes would run afoul of this. This code should prevent the BTree from overflowing in the first place, no? It should. But it failed to recognize that a Python int on a 64bit platform is too large to fit into an IITreeSet. For some reason this problem only started showing up after the BTree fixes. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Sun, Sep 19, 2010 at 1:50 AM, Nathan Van Gheem vangh...@gmail.com wrote: I'm experiencing some very frustrating issues with ZODB blob storage and am wondering if anyone has a pointer for me here. Basically, if we try to migrate a lot of our files to blobs or import/move large amounts of blob files, we'll get this error: invalidations out of order, None '\x03\x88\xf8c^\x8eG\xdd' I've found http://plone.293351.n2.nabble.com/plone-app-blob-migration-ARFilePreview-and-AROfficeTransforms-in-a-Plone-3-x-ZEO-cluster-sharing-they-td4291640.html and https://bugs.launchpad.net/zodb/+bug/509801 I'm running ZODB 3.8.5 on the zeo clients with Plone and Zope 2.10.11 and ZODB 3.9.5 on the zeoserver(we're running ZRS which requires ZODB 3.9 series--that's why it's different). I had thought the once we moved to ZODB 3.9.5 that this problem would disappear according the the resources I found; however, it seems that they are still present--turns out it's a client problem... We can't move the clients to zodb 3.9.5 since Zope 2.10.11 is not compatible with it. The fix for https://bugs.launchpad.net/zodb/+bug/509801 hasn't been backported to 3.9, although I assume it could be. Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Anyways, I was wondering if anyone has had some suggested resolutions to the problem. Is it safe to drop the client's zodb version down to 3.8.4 so where the problem doesn't seem to exist? Probably, but ... Or is it possible to backport the fix to a 3.8.6 release perhaps? I'm not entirely familiar with the zodb code, but I'd be willing to do the work if that was the only option. I do think it's practical to backport the fix. The hardest part would be resisting the temptation to backport other fixes. :) I'll look at making a 3.8 release with just this, or maybe this and a few other fixes if they are extremely low-hanging. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton j...@zope.com wrote: Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still claimed to support or depend on versions. We only cleaned out all those places in the Zope 2.12 code. Of course it's possible to create a branch of Zope 2.10 and backport these changes, but since 2.10 is long out of maintenance, those changes would never be released in an official version. Instead we recommend all Plone people to upgrade to Plone 4 which includes Zope 2.12 and ZODB 3.9 and even runs fine with ZODB 3.10. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 1:06 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton j...@zope.com wrote: Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still claimed to support or depend on versions. We only cleaned out all those places in the Zope 2.12 code. Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Mon, Sep 20, 2010 at 1:06 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton j...@zope.com wrote: Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still claimed to support or depend on versions. We only cleaned out all those places in the Zope 2.12 code. Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? I don't think - we encountered issues with some old storage still using versions while packing. Removing the versions from the storage fixed that problem. Andreas Or is this a matter of test failures? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope Plone D-72070 Tübingen| Produce Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMl6SsAAoJEADcfz7u4AZjsTILvi2HOy+gssyYwQxcG6fWPGQm en3FJZUlQO9BgBEdVvtoRD1ZG0GCt6t7eCCgy7mnt/rTm6qczkukWgCXHu97A0ww YnTdiIJ+VO8FWeIFX1dj6WQr9VElsDaWRVBUbzAU7iqAd0KpB7GrMWb1yKbzac8S RkLh2hPtlFezD1ZTP9yAecXMNML5Jv6Hr0eOv16e9Rxpvk9YT2j9y+6u9Y/Knc3O UFwWXYLNj1ZqOobH2IiM0ExS15VfB29L5hqz+9UuPfhIBV6lGqXf18kwD0Js0v0P GRY5llQA0LgcUa7p6QjglKOygrSqizM3hHlilqkfOnzVuGpLFwaSvZQ02myxDhnR nFS2v01c3AjAfnO/vej3RM/RssZ+q5WcnJQx62/TkZwjB7J+t72gXBQm8BqhocHy aVmeORNdJy9nSszZjh1S1Va1JuJaMIlIut4tD9lOp85stxBRzdgAWiQyAkVsNcyS PJp8SLCRBS+Vt8s1sdOtQ9FXV811EZ0= =8A8g -END PGP SIGNATURE- attachment: lists.vcf___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton j...@zope.com wrote: Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Last time I checked it did break. There's at least the temporary storage implementation that required changes to deal with the dropped version argument from several API's. But IIRC there's been more stuff in the control panel and startup code. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
Instead we recommend all Plone people to upgrade to Plone 4 which includes Zope 2.12 and ZODB 3.9 and even runs fine with ZODB 3.10. Unfortunately that is not always as easy as it sounds. We're pretty close to deploying here and the client doesn't want to deal with an major upgrade at this point. I'll look at making a 3.8 release with just this, or maybe this and a few other fixes if they are extremely low-hanging. That would be greatly appreciated. Let me know if there is anything I can do to help... Thanks, Nathan On Mon, Sep 20, 2010 at 1:25 PM, Hanno Schlichting ha...@hannosch.euwrote: On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton j...@zope.com wrote: Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Last time I checked it did break. There's at least the temporary storage implementation that required changes to deal with the dropped version argument from several API's. But IIRC there's been more stuff in the control panel and startup code. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
