Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 4:45 PM, Nathan Van Gheem wrote: > Also, I haven't seen any BTree issues yet. Is there anything that > would trigger this? We are running a 64 bit platform. Before this BTree fix, you could store integers in an IxThingy (or xIThingy) that were too large to fit in a 32-bit C variable. This would cause the values to overflow and be silently stored incorrectly. With the fix, a TypeError is raised when an attempt is made to store a value that is too large. Whether this fix will adversely affect your application, I don't know. Hanno seems to think it will. :) Avoiding silently storing values incorrectly looks all good to me. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
Also, I haven't seen any BTree issues yet. Is there anything that would trigger this? We are running a 64 bit platform. On Tue, Sep 21, 2010 at 3:44 PM, Nathan Van Gheem wrote: > Both releases will work for us I believe. We're currently on 3.8.5 > with a few custom fixes of my own so this release fits perfectly. > > Thank you for your help! > > On Tue, Sep 21, 2010 at 2:21 PM, Jim Fulton wrote: >> On Tue, Sep 21, 2010 at 11:23 AM, Hanno Schlichting >> wrote: >>> On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton wrote: > It was changes like this one > http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 I don't understand how the BTree changes would run afoul of this. This code should prevent the BTree from overflowing in the first place, no? >>> >>> It should. But it failed to recognize that a Python int on a 64bit >>> platform is too large to fit into an IITreeSet. For some reason this >>> problem only started showing up after the BTree fixes. >> >> Well, Hmph. >> >> I guess if Nathan can't run 3.8.6 (or 3.9), 3.8.4 should be OK, as the >> fix is for a false positive of a problem. >> >> Jim >> >> -- >> Jim Fulton >> > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
Both releases will work for us I believe. We're currently on 3.8.5 with a few custom fixes of my own so this release fits perfectly. Thank you for your help! On Tue, Sep 21, 2010 at 2:21 PM, Jim Fulton wrote: > On Tue, Sep 21, 2010 at 11:23 AM, Hanno Schlichting wrote: >> On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton wrote: It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 >>> >>> I don't understand how the BTree changes would run afoul of this. >>> This code should prevent the BTree from overflowing in the first place, no? >> >> It should. But it failed to recognize that a Python int on a 64bit >> platform is too large to fit into an IITreeSet. For some reason this >> problem only started showing up after the BTree fixes. > > Well, Hmph. > > I guess if Nathan can't run 3.8.6 (or 3.9), 3.8.4 should be OK, as the > fix is for a false positive of a problem. > > Jim > > -- > Jim Fulton > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 11:23 AM, Hanno Schlichting wrote: > On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton wrote: >>> It was changes like this one >>> http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 >> >> I don't understand how the BTree changes would run afoul of this. >> This code should prevent the BTree from overflowing in the first place, no? > > It should. But it failed to recognize that a Python int on a 64bit > platform is too large to fit into an IITreeSet. For some reason this > problem only started showing up after the BTree fixes. Well, Hmph. I guess if Nathan can't run 3.8.6 (or 3.9), 3.8.4 should be OK, as the fix is for a false positive of a problem. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 9:48 AM, Jim Fulton wrote: ... > I'm uncomfortable with with people using 3.8 with blobs in production. > I'm going to spend some time today seeing if I can add back enough > version api (as opposed to support :) in 3.9 to make it work with > Zope 2.10. Well, that was a waste of time. There seem to be a lot of issues, unrelated to versions afaict. I give up. :) Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton wrote: >> It was changes like this one >> http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 > > I don't understand how the BTree changes would run afoul of this. > This code should prevent the BTree from overflowing in the first place, no? It should. But it failed to recognize that a Python int on a 64bit platform is too large to fit into an IITreeSet. For some reason this problem only started showing up after the BTree fixes. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 11:16 AM, Hanno Schlichting wrote: > On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton wrote: >> On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting >> wrote: >>> Note that the BTrees 32bit bug fix (OverflowError to TypeError) also >>> needs some adjustments in Zope2 (IIRC in the DateIndex code) and some >>> add-ons (like CMFEditions). So it's not straightforward to use this in >>> older Plone versions. >> >> AFAIK, this fix doesn't convert overflow errors to type errors. It >> simply raises >> type errors in situations where data were simply stored incorrectly before. >> I'm surprised that this would cause problems. > > It was changes like this one > http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 I don't understand how the BTree changes would run afoul of this. This code should prevent the BTree from overflowing in the first place, no? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton wrote: > On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting wrote: >> Note that the BTrees 32bit bug fix (OverflowError to TypeError) also >> needs some adjustments in Zope2 (IIRC in the DateIndex code) and some >> add-ons (like CMFEditions). So it's not straightforward to use this in >> older Plone versions. > > AFAIK, this fix doesn't convert overflow errors to type errors. It > simply raises > type errors in situations where data were simply stored incorrectly before. > I'm surprised that this would cause problems. It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442&r1=115279&r2=115442 > BTW, if there a standard way to get Zope 2.10 to use a version of > ZODB other than the one that ships with it? It's all just a PYTHONPATH in the end. So if you put something on the path by the name of ZODB first, then the one in the software home inside Zope2 won't be loaded. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting wrote: > On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton wrote: >> 3.8.6 is available now with this bug fix and a few others. > > Note that the BTrees 32bit bug fix (OverflowError to TypeError) also > needs some adjustments in Zope2 (IIRC in the DateIndex code) and some > add-ons (like CMFEditions). So it's not straightforward to use this in > older Plone versions. AFAIK, this fix doesn't convert overflow errors to type errors. It simply raises type errors in situations where data were simply stored incorrectly before. I'm surprised that this would cause problems. > Personally I used 3.8.4 in all Plone 3 versions > with blobs. We've run this combination on large data sets without any > problems and I know of quite a number of other Plone companies having > done the same. For the most part, the work I did in ZODB 3.9 was motivated by a desire to get blobs to the point that I'd be willing to use them in production. >> I'm uncomfortable with with people using 3.8 with blobs in production. >> I'm going to spend some time today seeing if I can add back enough >> version api (as opposed to support :) in 3.9 to make it work with >> Zope 2.10. > > That's a very noble goal :) BTW, if there a standard way to get Zope 2.10 to use a version of ZODB other than the one that ships with it? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton wrote: > 3.8.6 is available now with this bug fix and a few others. Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So it's not straightforward to use this in older Plone versions. Personally I used 3.8.4 in all Plone 3 versions with blobs. We've run this combination on large data sets without any problems and I know of quite a number of other Plone companies having done the same. > I'm uncomfortable with with people using 3.8 with blobs in production. > I'm going to spend some time today seeing if I can add back enough > version api (as opposed to support :) in 3.9 to make it work with > Zope 2.10. That's a very noble goal :) Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Tue, Sep 21, 2010 at 12:25 AM, Nathan Van Gheem wrote: ... >> I'll look at making a 3.8 release with just this, or maybe this and >> >> a few other fixes if they are extremely low-hanging. > > That would be greatly appreciated. Let me know if there is anything I can do > to help... 3.8.6 is available now with this bug fix and a few others. I'll be making a 3.9.6 release later today. I'm uncomfortable with with people using 3.8 with blobs in production. I'm going to spend some time today seeing if I can add back enough version api (as opposed to support :) in 3.9 to make it work with Zope 2.10. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
> > Instead we recommend all Plone people to upgrade to Plone 4 which includes Zope 2.12 and ZODB 3.9 and even runs fine with ZODB 3.10. Unfortunately that is not always as easy as it sounds. We're pretty close to deploying here and the client doesn't want to deal with an major upgrade at this point. I'll look at making a 3.8 release with just this, or maybe this and a few other fixes if they are extremely low-hanging. That would be greatly appreciated. Let me know if there is anything I can do to help... Thanks, Nathan On Mon, Sep 20, 2010 at 1:25 PM, Hanno Schlichting wrote: > On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton wrote: > > Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use > > versions? Or is this a matter of test failures? > > Last time I checked it did break. There's at least the temporary > storage implementation that required changes to deal with the dropped > version argument from several API's. But IIRC there's been more stuff > in the control panel and startup code. > > Hanno > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton wrote: > Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use > versions? Or is this a matter of test failures? Last time I checked it did break. There's at least the temporary storage implementation that required changes to deal with the dropped version argument from several API's. But IIRC there's been more stuff in the control panel and startup code. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Mon, Sep 20, 2010 at 1:06 PM, Hanno Schlichting wrote: >> On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton wrote: >>> Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm >>> curious why? >> Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. >> But in 3.9 versions got removed. There was a good deal of code in Zope >> 2 that still claimed to support or depend on versions. We only cleaned >> out all those places in the Zope 2.12 code. > > Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use > versions? I don't think - we encountered issues with some old storage still using versions while packing. Removing the versions from the storage fixed that problem. Andreas Or is this a matter of test failures? > > Jim > > -- > Jim Fulton > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope & Plone D-72070 Tübingen| Produce & Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope & Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMl6SsAAoJEADcfz7u4AZjsTILvi2HOy+gssyYwQxcG6fWPGQm en3FJZUlQO9BgBEdVvtoRD1ZG0GCt6t7eCCgy7mnt/rTm6qczkukWgCXHu97A0ww YnTdiIJ+VO8FWeIFX1dj6WQr9VElsDaWRVBUbzAU7iqAd0KpB7GrMWb1yKbzac8S RkLh2hPtlFezD1ZTP9yAecXMNML5Jv6Hr0eOv16e9Rxpvk9YT2j9y+6u9Y/Knc3O UFwWXYLNj1ZqOobH2IiM0ExS15VfB29L5hqz+9UuPfhIBV6lGqXf18kwD0Js0v0P GRY5llQA0LgcUa7p6QjglKOygrSqizM3hHlilqkfOnzVuGpLFwaSvZQ02myxDhnR nFS2v01c3AjAfnO/vej3RM/RssZ+q5WcnJQx62/TkZwjB7J+t72gXBQm8BqhocHy aVmeORNdJy9nSszZjh1S1Va1JuJaMIlIut4tD9lOp85stxBRzdgAWiQyAkVsNcyS PJp8SLCRBS+Vt8s1sdOtQ9FXV811EZ0= =8A8g -END PGP SIGNATURE- <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 1:06 PM, Hanno Schlichting wrote: > On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton wrote: >> Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm >> curious why? > > Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. > But in 3.9 versions got removed. There was a good deal of code in Zope > 2 that still claimed to support or depend on versions. We only cleaned > out all those places in the Zope 2.12 code. Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton wrote: > Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm > curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still claimed to support or depend on versions. We only cleaned out all those places in the Zope 2.12 code. Of course it's possible to create a branch of Zope 2.10 and backport these changes, but since 2.10 is long out of maintenance, those changes would never be released in an official version. Instead we recommend all Plone people to upgrade to Plone 4 which includes Zope 2.12 and ZODB 3.9 and even runs fine with ZODB 3.10. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB: invalidations out of order
On Sun, Sep 19, 2010 at 1:50 AM, Nathan Van Gheem wrote: > I'm experiencing some very frustrating issues with ZODB blob storage and am > wondering if anyone has a pointer for me here. > Basically, if we try to migrate a lot of our files to blobs or import/move > large amounts of blob files, we'll get this error: >> >> invalidations out of order, None < '\x03\x88\xf8c^\x8eG\xdd' > > I've > found http://plone.293351.n2.nabble.com/plone-app-blob-migration-ARFilePreview-and-AROfficeTransforms-in-a-Plone-3-x-ZEO-cluster-sharing-they-td4291640.html and https://bugs.launchpad.net/zodb/+bug/509801 > I'm running ZODB 3.8.5 on the zeo clients with Plone and Zope 2.10.11 and > ZODB 3.9.5 on the zeoserver(we're running ZRS which requires ZODB 3.9 > series--that's why it's different). I had thought the once we moved to ZODB > 3.9.5 that this problem would disappear according the the resources I found; > however, it seems that they are still present--turns out it's a client > problem... We can't move the clients to zodb 3.9.5 since Zope 2.10.11 is not > compatible with it. The fix for https://bugs.launchpad.net/zodb/+bug/509801 hasn't been backported to 3.9, although I assume it could be. Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? > Anyways, I was wondering if anyone has had some suggested resolutions to the > problem. Is it safe to drop the client's zodb version down to 3.8.4 so where > the problem doesn't seem to exist? Probably, but ... > Or is it possible to backport the fix to > a 3.8.6 release perhaps? I'm not entirely familiar with the zodb code, but > I'd be willing to do the work if that was the only option. I do think it's practical to backport the fix. The hardest part would be resisting the temptation to backport other fixes. :) I'll look at making a 3.8 release with just this, or maybe this and a few other fixes if they are extremely low-hanging. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB: invalidations out of order
I'm experiencing some very frustrating issues with ZODB blob storage and am wondering if anyone has a pointer for me here. Basically, if we try to migrate a lot of our files to blobs or import/move large amounts of blob files, we'll get this error: invalidations out of order, None < '\x03\x88\xf8c^\x8eG\xdd' I've found http://plone.293351.n2.nabble.com/plone-app-blob-migration-ARFilePreview-and-AROfficeTransforms-in-a-Plone-3-x-ZEO-cluster-sharing-they-td4291640.html and https://bugs.launchpad.net/zodb/+bug/509801 I'm running ZODB 3.8.5 on the zeo clients with Plone and Zope 2.10.11 and ZODB 3.9.5 on the zeoserver(we're running ZRS which requires ZODB 3.9 series--that's why it's different). I had thought the once we moved to ZODB 3.9.5 that this problem would disappear according the the resources I found; however, it seems that they are still present--turns out it's a client problem... We can't move the clients to zodb 3.9.5 since Zope 2.10.11 is not compatible with it. Anyways, I was wondering if anyone has had some suggested resolutions to the problem. Is it safe to drop the client's zodb version down to 3.8.4 so where the problem doesn't seem to exist? Or is it possible to backport the fix to a 3.8.6 release perhaps? I'm not entirely familiar with the zodb code, but I'd be willing to do the work if that was the only option. Thanks, Nathan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
