Re: [Zope3-dev] buildbot failure in Zope3 trunk 2.4 FreeBSD tmiddleton
[Christian Theune] Hah. Looks like BTrees can accept 2**31 on machines where maxint is a larger ... Note that accept in this case means silently loses the most-significant bits, and that's a BTree bug on platforms where sizeof(long) sizeof(int): http://www.zope.org/Collectors/Zope/1592 2**31 doesn't actually lose any bits when you store it, but it will probably misinterpret the high-order data bit as the sign bit when you fetch it again, magically changing it into -2**31. As we hardcoded the search space to 2**31, I made the test check whether btrees accept the number or not and differ in the expected results. Should work on that 64bit machine now too. Did you see my simpler suggestion for fixing the underlying bug (it was a one-liner change to the original code)? When you get tired of fighting the 64-bit BTree bug here (it will be a minor miracle if the test actually passes now on a 64-bit box, despite all you've tried ...), look that up ;-) ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] buildbot failure in Zope3 trunk 2.4 FreeBSD tmiddleton
Hi, thanks Tim, for the partial enlightenment. :) Unfortunately my BTree/C-wisdom is much much smaller than yours, so I got to check a couple of assumptions over here. :) Tim Peters wrote: [Christian Theune] Hah. Looks like BTrees can accept 2**31 on machines where maxint is a larger ... Note that accept in this case means silently loses the most-significant bits, and that's a BTree bug on platforms where sizeof(long) sizeof(int): http://www.zope.org/Collectors/Zope/1592 I was able to reproduce this on an Intel 64-Bit machine (EM64T) running Linux and gcc. For one: I didn't see any compiler warnings. That sounds bad, right? 2**31 doesn't actually lose any bits when you store it, but it will probably misinterpret the high-order data bit as the sign bit when you fetch it again, magically changing it into -2**31. I can store 2**31 in the BTree, but the keys() method will tell you that it actually stored -2**31. As we hardcoded the search space to 2**31, I made the test check whether btrees accept the number or not and differ in the expected results. Should work on that 64bit machine now too. Did you see my simpler suggestion for fixing the underlying bug (it was a one-liner change to the original code)? When you get tired of fighting the 64-bit BTree bug here (it will be a minor miracle if the test actually passes now on a 64-bit box, despite all you've tried ...), look that up ;-) Nope, I didn't find your one-liner. :) Can you post it explicitly for my blind eyes? I think I could have made up something that made it work, but I started looking into making the BTree behave sanely. My idea was, to modify the BTree code in a way that it actually checks after the type cast whether the casted value is equal to the requested key, or alternatively try making the CHECK_KEY macro do an exact type match instead of allowing subclasses. But that wouldn't work either as 2**31 is still an int on those platforms. I'm a bit puzzled now. :) Christian -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] buildbot failure in Zope3 trunk 2.4 FreeBSD tmiddleton
[Christian] thanks Tim, for the partial enlightenment. :) Unfortunately my BTree/C-wisdom is much much smaller than yours, so I got to check a couple of assumptions over here. :) Yup, it really helps if you have a 64-bit box to check them on. [Christian] Hah. Looks like BTrees can accept 2**31 on machines where maxint is a larger ... [Tim] Note that accept in this case means silently loses the most-significant bits, and that's a BTree bug on platforms where sizeof(long) sizeof(int): http://www.zope.org/Collectors/Zope/1592 I was able to reproduce this on an Intel 64-Bit machine (EM64T) running Linux and gcc. And I expect the same will happen on any 64-bit platform other than Win64 (which is unique, AFAICT, in leaving sizeof(long) == 4 instead of boosting it to 8). For one: I didn't see any compiler warnings. That sounds bad, right? The underlying BTree bug is assignments of the form (usually hidden in macro expansions): some_C_int = some_C_long; When sizeof(int) sizeof(long), that can silently lose information. I was really hoping that major compilers on boxes where sizeof(int) sizeof(long) would warn about that. Oh well. The problem arises because what _Python_ calls int is what C calls long, but the I-flavor BTree code stores C int, and C int doesn't correspond to any Python type (except by accident on 32-bit boxes). C int is 4 bytes on all known current 32- and 64-bit platforms, but the size of what Python calls int varies. The BTree code isn't aware of the possible mismatch, storing Python int (C long) into C int without any checking. 2**31 doesn't actually lose any bits when you store it, but it will probably misinterpret the high-order data bit as the sign bit when you fetch it again, magically changing it into -2**31. I can store 2**31 in the BTree, but the keys() method will tell you that it actually stored -2**31. Right. If it's not clear, this is because the _bit_ pattern 0x8000 is 2**31 when viewed as an 8-byte C long, but is -2**31 when viewed as a 4-byte C long. If you had, e.g., stored 2**32 instead, you would have gotten 0 back when you fetched it (the top 32 bits are simply thrown away). ... Did you see my simpler suggestion for fixing the underlying bug (it was a one-liner change to the original code)? When you get tired of fighting the 64-bit BTree bug here (it will be a minor miracle if the test actually passes now on a 64-bit box, despite all you've tried ...), look that up ;-) Nope, I didn't find your one-liner. :) Can you post it explicitly for my blind eyes? Here; it was a reply to one of the checkin messages: http://mail.zope.org/pipermail/checkins/2006-June/002395.html The key problem in the original intid code is that it used randint() instead of randrange(), theoretically allowing 2**31 to be a return value. It's _almost_ enough just to use randrange() instead. Unfortunately, that's not quite enough; see the msg for what is. It _could_ be fixed by adding 2 characters to the original, changing 2**31 to 2**31-2 (or to 0x7ffe), but that would leave it pretty mysterious ;-) I think I could have made up something that made it work, but I started looking into making the BTree behave sanely. My idea was, to modify the BTree code in a way that it actually checks after the type cast whether the casted value is equal to the requested key, or alternatively try making the CHECK_KEY macro do an exact type match instead of allowing subclasses. But that wouldn't work either as 2**31 is still an int on those platforms. I'm a bit puzzled now. :) There's potential silent information loss for both Ix-flavor BTree keys and xI-flavor BTree values. There are two robust ways to check (robust means that Python's C code has used these ways at various times for many years now without problems). Complain if: some_C_long INT_MIN || some_C_long INT_MAX or complain if (this sounds close to what you had in mind above, and is my favorite): (long)(int)some_C_long != some_C_long Because the problem is due to bogus assumptions about the relationship between C types, it's not going to help to examine Python's idea of type. Checking isn't needed (can't fail) if SIZEOF_INT == SIZEOF_LONG (Python.h supplies definitions for those macros), so there's some worth to skipping checks when that's not true. Unfortunately, C doesn't allow #if preprocessor statements _inside_ macro expansions, so the best way to do that isn't immediately clear. In short, irritiating little issues abound :-(. That's why I couldn't make time to fix it (relatively high cost with no benefit on most Zope platforms). Note that if ZODB moves to 64-bit Ix/xI BTrees on all boxes (IIRC, Jim and Fred were agitating in that direction, but suffered massive short-sighted ;-) opposition), the BTree problem would go away by magic (C int would no longer be the type of Ix keys or xI values). ___ Zope3-dev mailing list
Re: [Zope3-dev] buildbot failure in Zope3 trunk 2.4 FreeBSD tmiddleton
Thanks for the clarification, Tim! Due to the possible future use of 64-bit BTrees, I'm inclined to go for the 2-character fix for now and possibly remove it once we hit 64-bit everywhere. Otherwise I'd agree on the fix using the comparison you mentioned: (long)(int)some_C_long != some_C_long (That would require me to activate my very rusty C-knowledge and to change and possibly break things in weird ways.) So, could somebody (Fred? Jim?) jump in here and tell me which fix they are fine to use? Christian Tim Peters wrote: [Christian] thanks Tim, for the partial enlightenment. :) Unfortunately my BTree/C-wisdom is much much smaller than yours, so I got to check a couple of assumptions over here. :) Yup, it really helps if you have a 64-bit box to check them on. [Christian] Hah. Looks like BTrees can accept 2**31 on machines where maxint is a larger ... [Tim] Note that accept in this case means silently loses the most-significant bits, and that's a BTree bug on platforms where sizeof(long) sizeof(int): http://www.zope.org/Collectors/Zope/1592 I was able to reproduce this on an Intel 64-Bit machine (EM64T) running Linux and gcc. And I expect the same will happen on any 64-bit platform other than Win64 (which is unique, AFAICT, in leaving sizeof(long) == 4 instead of boosting it to 8). For one: I didn't see any compiler warnings. That sounds bad, right? The underlying BTree bug is assignments of the form (usually hidden in macro expansions): some_C_int = some_C_long; When sizeof(int) sizeof(long), that can silently lose information. I was really hoping that major compilers on boxes where sizeof(int) sizeof(long) would warn about that. Oh well. The problem arises because what _Python_ calls int is what C calls long, but the I-flavor BTree code stores C int, and C int doesn't correspond to any Python type (except by accident on 32-bit boxes). C int is 4 bytes on all known current 32- and 64-bit platforms, but the size of what Python calls int varies. The BTree code isn't aware of the possible mismatch, storing Python int (C long) into C int without any checking. 2**31 doesn't actually lose any bits when you store it, but it will probably misinterpret the high-order data bit as the sign bit when you fetch it again, magically changing it into -2**31. I can store 2**31 in the BTree, but the keys() method will tell you that it actually stored -2**31. Right. If it's not clear, this is because the _bit_ pattern 0x8000 is 2**31 when viewed as an 8-byte C long, but is -2**31 when viewed as a 4-byte C long. If you had, e.g., stored 2**32 instead, you would have gotten 0 back when you fetched it (the top 32 bits are simply thrown away). ... Did you see my simpler suggestion for fixing the underlying bug (it was a one-liner change to the original code)? When you get tired of fighting the 64-bit BTree bug here (it will be a minor miracle if the test actually passes now on a 64-bit box, despite all you've tried ...), look that up ;-) Nope, I didn't find your one-liner. :) Can you post it explicitly for my blind eyes? Here; it was a reply to one of the checkin messages: http://mail.zope.org/pipermail/checkins/2006-June/002395.html The key problem in the original intid code is that it used randint() instead of randrange(), theoretically allowing 2**31 to be a return value. It's _almost_ enough just to use randrange() instead. Unfortunately, that's not quite enough; see the msg for what is. It _could_ be fixed by adding 2 characters to the original, changing 2**31 to 2**31-2 (or to 0x7ffe), but that would leave it pretty mysterious ;-) I think I could have made up something that made it work, but I started looking into making the BTree behave sanely. My idea was, to modify the BTree code in a way that it actually checks after the type cast whether the casted value is equal to the requested key, or alternatively try making the CHECK_KEY macro do an exact type match instead of allowing subclasses. But that wouldn't work either as 2**31 is still an int on those platforms. I'm a bit puzzled now. :) There's potential silent information loss for both Ix-flavor BTree keys and xI-flavor BTree values. There are two robust ways to check (robust means that Python's C code has used these ways at various times for many years now without problems). Complain if: some_C_long INT_MIN || some_C_long INT_MAX or complain if (this sounds close to what you had in mind above, and is my favorite): (long)(int)some_C_long != some_C_long Because the problem is due to bogus assumptions about the relationship between C types, it's not going to help to examine Python's idea of type. Checking isn't needed (can't fail) if SIZEOF_INT == SIZEOF_LONG (Python.h supplies definitions for those macros), so there's some worth to skipping checks when that's not true. Unfortunately, C doesn't allow #if preprocessor statements _inside_ macro expansions, so the
Re: [Zope3-dev] buildbot failure in Zope3 trunk 2.4 FreeBSD tmiddleton
[Christian] Thanks for the clarification, Tim! You're welcome :-) Due to the possible future use of 64-bit BTrees, I'm inclined to go for the 2-character fix for now and possibly remove it once we hit 64-bit everywhere. The BTree bug and the intid bug are quite distinct. intid should be repaired regardless, and is easy to repair. The only reason the BTree bug got sucked into this is because an overly-elaborate way of repairing the intid bug bashed into the BTree bug (the elaborate initd fix reasonably assumed that Ix BTrees would complain if you tried to store a key they can't actually handle, but in fact they don't always complain). All the _simple_ ways of repairing intid avoid the BTree bug. In a 64-bit-everywhere world, presumably intid will want to change to use an id range substantially larger than 31 bits anyway. BTW, note that the simple intid fixes are essentially untestable, and for the same reason you can't actually write a convincing test now that _shows_ that the current (pre-patch) intid is buggy. There are only 2 return values (out of 2**31+1 current possible outcomes) that create a problem, so it's a one-in-a-billion chance of screwing up. The simple fixes just remove (exactly) those two problem cases. It's trivial to prove that the current intid has that bug, and substantially harder to prove that the 2-character fix repairs it (but trivial to prove that my original suggestion repairs it), but it's very much harder to write a test showing either the bug or its absence. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: SVN: zope.testing/branches/stub-testSetUp_in_layer/src/zope/testing/testrunner Implement layer.testSetUp and layer.testTearDown support
Stuart Bishop wrote: Log message for revision 68689: +def testSetUp(self): +A layer may define a setup method to be called before each +individual test. + +for layer in reversed(self.layers): +if hasattr(layer, 'testSetUp'): +layer.testSetUp() Ah cool, pretty useful thanks. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: SVN: Zope3/branches/3.3/ - fix for issue 517: intid might generate long instead of int which conflicts with btrees
Christian Theune wrote: Log message for revision 68700: --- Zope3/branches/3.3/src/zope/app/intid/__init__.py 2006-06-17 03:29:43 UTC (rev 68699) +++ Zope3/branches/3.3/src/zope/app/intid/__init__.py 2006-06-17 04:10:29 UTC (rev 68700) @@ -47,7 +47,10 @@ implements(IIntIds) -_v_nextid = None +_v_nextid = None + +# Used for testability of random function +__randint__ = random.randint You should avoid such __names__, they are supposed to be reserved. Here, __randint would have been quite enough I believe. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Content-types in ResourceDirectory
Still playing with Dojo. Very Nice! :) I'm having a problem with content-type for items served from a resourceDirectory. In particular, one piece of dojo, a file with a .html extension begins with ?xml version=1.0 encoding=UTF-8?. The DOCTYPE says XHTML 1.0 Strict, but Zope3 returns it as text/plain (X-Content-Type-Warning: guessed from content). Unfortunately, that is a problem, because the client looks at content-type to decide how to parse. If I set the content-type to text/html, it works OK. So, is this a Dojo bug or a Zope3 bug? I've done a wsgi filter trick to get around this for now. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 trunk 2.4 Linux tlotze
The Buildbot has detected a failed build of Zope3 trunk 2.4 Linux tlotze. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 6215 Blamelist: ctheune BUILD FAILED: failed svn sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
