Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Tim Peters [EMAIL PROTECTED] wrote: This is a wrong time in the release process to take on chance on discovering a flaky LONG_MIN on some box, so I want to keep the code as much as possible like what's already there (which worked fine for 10 years on all known boxes) for now. No, it didn't. I reported a bug a couple of years back. A blanket rule not to use symbols is clearly wrong, but there are good reasons not to want to rely on LONG_MIN (or INT_MIN for that matter). Because of some incredibly complex issues (which I only know some of), it hasn't been consistently -(1+LONG_MAX) on twos' complement machines. There are good reasons for making it -LONG_MAX, but they aren't the ones that actually cause it to be so. There are, however, very good reasons for using BOTH tests. I.e. if I have a C system which defines LONG_MIN to be -LONG_MAX because it uses -(1+LONG_MAX) for an integer NaN indicator in some contexts, you really DON'T want to create such a value. I don't know if there are any such C systems, but there have been other languages that did. I hope that Guido wasn't saying that Python should EVER rely on signed integer overflow wrapping in twos' complement. Despite the common usage, Java and all that, it is perhaps the worst systematic architectural change to have happened in 30 years, and accounts for a good 30% of all hard bugs in many classes of program. Simple buffer overflow is fairly easy to avoid by good programming style; integer overflow causing trashing of unpredictable data isn't. Any decent programming language (like Python!) regards integer overflow as an error, and the moves to make C copy Java semantics are yet another step away from software engineering in the direction of who-gives-a-damn hacking. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Hi Tim, On Sat, Aug 26, 2006 at 08:37:46PM -0400, Tim Peters wrote: [Thomas Wouters] Why not just ... x == LONG_MIN? it's better (when possible) not to tie the code to that `x` was specifically declared as type long (e.g., just more stuff that will break if Python decides to make its short int of type PY_LONG_LONG instead). The proposed correct fix breaks this goal too: if (y == -1 x 0 (unsigned long)x == -(unsigned long)x). ^^ A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
On Wednesday 30 August 2006 08:57, Tim Peters wrote: Speaking of which, I saw no feedback on the proposed patch in http://mail.python.org/pipermail/python-dev/2006-August/068502.html so I'll just check that in tomorrow. Fine with me! thanks, Anthony -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
On Wednesday 30 August 2006 08:57, Tim Peters wrote: Speaking of which, I saw no feedback on the proposed patch in http://mail.python.org/pipermail/python-dev/2006-August/068502.html so I'll just check that in tomorrow. This should also be backported to release24-maint and release23-maint. Let me know if you can't do the backport... -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
I believe some of the others here might be interested in some other postings on the gcc mailing list regarding this issue (which weren't cross-posted here)... http://gcc.gnu.org/ml/gcc/2006-08/msg00516.html http://gcc.gnu.org/ml/gcc/2006-08/msg00517.html It makes clear that the impact of this change in gcc was considered and the reasoning on their decision to follow the standard so closely. http://gcc.gnu.org/ml/gcc/2005-06/msg01238.html Just so other beside Guido see those. Jack ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Jack Howarth wrote: I believe some of the others here might be interested in some other postings on the gcc mailing list regarding this issue (which weren't cross-posted here)... http://gcc.gnu.org/ml/gcc/2006-08/msg00516.html http://gcc.gnu.org/ml/gcc/2006-08/msg00517.html It makes clear that the impact of this change in gcc was considered and the reasoning on their decision to follow the standard so closely. http://gcc.gnu.org/ml/gcc/2005-06/msg01238.html Just so other beside Guido see those. This discussion just highlights how different the perspective of much of the C community (especially its language committee and compiler writers) is on error handling and language safety, relative to the perspective of the community that uses Python and other memory-safe languages. The C perspective is that programs have to be correct, and if they're not correct, it basically doesn't matter what happens. The safe language perspective is that of course we try to write correct programs, but if a program is not correct, then it *really matters* what the error behaviour is. Ideally, it should be something that facilitates debugging and that doesn't silently propagate an incorrect result. Undefined behaviour is right out. For example, from http://gcc.gnu.org/ml/gcc/2006-08/msg00522.html: # The gcc developers always face two competing constituencies: do not # change the compiler behaviour so that existing code which relies on # undefined behaviour continues to work, and do change the compiler # behaviour so that new code which does not rely on undefined behaviour # runs faster. In general, being compiler developers, we come down on # the side of making code which does not rely on undefined behaviour run # faster. ... and rarely even consider whether the behaviour *should* have been undefined or not. (There are many people who use C but do not agree with that perspective on error handling -- I'm one of them. These people often devise elaborate error handling frameworks, and compile at low optimization levels when it doesn't hurt performance.) -- David Hopwood [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
On Sunday 27 August 2006 05:06, Jack Howarth wrote: I discovered that gcc 4.2 exposes a flaw with signed integer overflows in python. This bug and the necessary fix has been discussed in detail on the gcc mailing list. I have filed a detailed bug report and the recommended patch proposed by the gcc developers. This problem should be addressed BEFORE python 2.5 is released. The bug report is... [ 1545668 ] gcc trunk (4.2) exposes a signed integer overflows in the python sourceforge bug tracker. Thanks in advance for attempting to fix this before Python 2.5 is released. Jack Regardless of whether we consider gcc's behaviour to be correct or not, I do agree we need a fix for this in 2.5 final. That should also be backported to release24-maint for the 2.4.4 release, and maybe release23-maint, as Barry recently started talking about cutting a 2.3.6. Can I nominate Tim, with his terrifying knowledge of C compiler esoterica, as the person to pick the best fix? Thanks, Anthony ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
[Anthony Baxter] Regardless of whether we consider gcc's behaviour to be correct or not, It is correct, but more to the point it's, umm, /there/ ;-) I do agree we need a fix for this in 2.5 final. That should also be backported to release24-maint for the 2.4.4 release, and maybe release23-maint, as Barry recently started talking about cutting a 2.3.6. Can I nominate Tim, with his terrifying knowledge of C compiler esoterica, as the person to pick the best fix? It's a bitch. Changing to if (y == -1 x 0 (unsigned long)x == -(unsigned long)x) is the obvious fix, but violates our no warnings policy: the MS compiler warns about applying unary minus to an unsigned operand -- it looks insane to /their/ compiler writers ;-). Elegant patch below -- LOL. Would be nice if someone verified it worked on a box where it matters. Would also be nice if people checked to see whether their compiler(s) warn about something else now. IIndex: Objects/intobject.c === --- Objects/intobject.c (revision 51618) +++ Objects/intobject.c (working copy) @@ -564,8 +564,14 @@ integer division or modulo by zero); return DIVMOD_ERROR; } - /* (-sys.maxint-1)/-1 is the only overflow case. */ - if (y == -1 x 0 x == -x) + /* (-sys.maxint-1)/-1 is the only overflow case. x is the most +* negative long iff x 0 and, on a 2's-complement box, x == -x. +* However, -x is undefined (by C) if x /is/ the most negative long +* (it's a signed overflow case), and some compilers care. So we cast +* x to unsigned long first. However, then other compilers warn about +* applying unary minus to an unsigned operand. Hence the weird 0-. +*/ + if (y == -1 x 0 (unsigned long)x == 0-(unsigned long)x) return DIVMOD_OVERFLOW; xdivy = x / y; xmody = x - xdivy * y; ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] gcc 4.2 exposes signed integer overflows
I discovered that gcc 4.2 exposes a flaw with signed integer overflows in python. This bug and the necessary fix has been discussed in detail on the gcc mailing list. I have filed a detailed bug report and the recommended patch proposed by the gcc developers. This problem should be addressed BEFORE python 2.5 is released. The bug report is... [ 1545668 ] gcc trunk (4.2) exposes a signed integer overflows in the python sourceforge bug tracker. Thanks in advance for attempting to fix this before Python 2.5 is released. Jack ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
On 8/26/06, Jack Howarth [EMAIL PROTECTED] wrote: I discovered that gcc 4.2 exposes a flaw with signed integer overflows in python. This bug and the necessary fix has been discussed in detail on the gcc mailing list. I have filed a detailed bug report and the recommended patch proposed by the gcc developers. This problem should be addressed BEFORE python 2.5 is released. The bug report is... [ 1545668 ] gcc trunk (4.2) exposes a signed integer overflows in the python sourceforge bug tracker. Thanks in advance for attempting to fix this before Python 2.5 is released. I'm not sure I follow why this isn't considered a regression in GCC. Clearly, on all current hardware, x == -x is also true for the most negative int (0x8000 on a 32-bit box). Why is GCC attempting to break our code (and then blaming us for it!) by using the C standard's weaselwords that signed integer overflow is undefined, despite that it has had a traditional meaning on 2's complement hardware for many decades? If GCC starts to enforce everything that the C standard says is undefined then very few programs will still work... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Guido, You'll never win that argument with the gcc developers. If you rely on undefined behavior in the c language standard, they have in the past, and will continue to, feel free to ignore those cases. If you plan on ignoring this issue, just be prepared to see a testcase failure in the python testsuite when python is built with gcc 4.2. Jack ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Guido van Rossum wrote: On 8/26/06, Jack Howarth [EMAIL PROTECTED] wrote: I discovered that gcc 4.2 exposes a flaw with signed integer overflows in python. This bug and the necessary fix has been discussed in detail on the gcc mailing list. I have filed a detailed bug report and the recommended patch proposed by the gcc developers. This problem should be addressed BEFORE python 2.5 is released. The bug report is... [ 1545668 ] gcc trunk (4.2) exposes a signed integer overflows in the python sourceforge bug tracker. Thanks in advance for attempting to fix this before Python 2.5 is released. I'm not sure I follow why this isn't considered a regression in GCC. Clearly, on all current hardware, x == -x is also true for the most negative int (0x8000 on a 32-bit box). Why is GCC attempting to break our code (and then blaming us for it!) by using the C standard's weaselwords that signed integer overflow is undefined, despite that it has had a traditional meaning on 2's complement hardware for many decades? If GCC starts to enforce everything that the C standard says is undefined then very few programs will still work... +1. It's clearly undefined behaviour, but that doesn't mean that it is a good idea for gcc to be making changes that could silently break a lot of code, in pursuit of a microoptimization that is unlikely to significantly help performance. CPython should be fixed anyway. The correct fix is if (y == -1 x 0 (unsigned long)x == -(unsigned long)x). -- David Hopwood [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
On 8/26/06, David Hopwood [EMAIL PROTECTED] wrote: CPython should be fixed anyway. The correct fix isif (y == -1 x 0 (unsigned long)x == -(unsigned long)x).Why not just ... x == LONG_MIN? -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Jack Howarth wrote: Guido, You'll never win that argument with the gcc developers. If you rely on undefined behavior in the c language standard, they have in the past, and will continue to, feel free to ignore those cases. If you plan on ignoring this issue, just be prepared to see a testcase failure in the python testsuite when python is built with gcc 4.2. In addition, i'm surprised that you don't see these failures on other compilers. I know XLC, for example, would treat signed integer overflow as undefined at all opt levels as well. I would not be surprised to find Intel's compiler doing the same thing. Speaking as a gcc optimization person, if we were to treat signed integer overflow as wrapping, you would inhibit a very large number of loop optimizations on a very large class of loops (due to a number of reasons, from no longer being able to prove termination of loops or estimation of number of iterations, to other things). Heck, XLC at -O3+ will ignore *unsigned* integer wraparound of loop indices unless you use -qstrict_induction. As the docs for this option say, using that option can cause severe performance degradation (I happen to know the numbers from when i worked at IBM, and they are surprisingly high). GCC is willing to take the performance hit relative to other compilers on things like that to be a standard conforming compiler, just like we are willing to optimize code based on the assumption that you do not invoke undefined behavior. --Dan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
[David Hopwood] CPython should be fixed anyway. The correct fix is if (y == -1 x 0 (unsigned long)x == -(unsigned long)x). Note that this was already suggested in the bug report. [Thomas Wouters] Why not just ... x == LONG_MIN? In full, if (y == -1 x == LONG_MIN) should work too. In practice we try to avoid numeric symbols from platform header files because so many platforms have screwed these up over the centuries (search for LONG_BIT or HUGE_VAL ;-)), and because it's better (when possible) not to tie the code to that `x` was specifically declared as type long (e.g., just more stuff that will break if Python decides to make its short int of type PY_LONG_LONG instead). In this specific case, there may also have been a desire to avoid generating a memory load for a fat constant. However, since this is integer division, in real life (outside the test suite) we'll never go beyond the y == -1 test. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Thomas Wouters wrote: On 8/26/06, David Hopwood [EMAIL PROTECTED] wrote: CPython should be fixed anyway. The correct fix is if (y == -1 x 0 (unsigned long)x == -(unsigned long)x). Why not just ... x == LONG_MIN? Because the intent is to check that x / y does not overflow a long, and x == LONG_MIN would not cause an overflow on 1's complement or sign-magnitude systems. (CPython has probably only been tested on 2's complement systems anyway, but if we're going to be pedantic about depending only on things in the C standard...) -- David Hopwood [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
[David Hopwood] (CPython has probably only been tested on 2's complement systems anyway, Definitely so. Are there any boxes using 1's-comp or sign-magnitude integers anymore? Python assumes 2's-comp in many places. but if we're going to be pedantic about depending only on things in the C standard...) No, in that respect we're driven by the silliest decisions made by C compiler writers ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
