Re: [avr-gcc-list] More results from the testsuite with avrtest
Weddington, Eric wrote: -Original Message- From: Paulo Marques [...] This gives us 13781 revisions to test. By bisecting, we should be able to lock in on the offending patch in less than 14 attempts. A script that automated this bisection process (checkout middle revision, test it, mark good/bad) would be great to find what exectly caused specific regressions. I may give it a try tomorrow, Please see the GCC project about this. IIRC, they already have a script that does a binary search on failing test cases. IIRC, it is in the contrib subdirectory. If I had a nickel for every time I come up with a nice original idea that someone has already thought of before ;) I looked into what is already done to help binary searches like I wanted to do, and there is a nice mechanism in place to solve one of the problems I was having: checking out tens or hundreds of different gcc revisions from the gcc repository over the internet :P The existing solution consists of rsync'ing the gcc repository into a local repository, so that you can checkout any revision locally. You pay an initial setup cost, but then you can checkout any revision for free :) The initial setup cost is 13Gb. Even if I'm lucky enough to get my internet connection to go at full speed, that's still at least 9 hours of download, so I guess I won't have any results tonight. I can't wait to get a bisecting environment ready ;) -- Paulo Marques Software Development Department - Grupo PIE, S.A. Phone: +351 252 290600, Fax: +351 252 290601 Web: www.grupopie.com For every problem there is one solution which is simple, neat, and wrong. H. L. Mencken ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Ok, here are my test results - same GCC as Paulo - except I have install some patches in avr to fix some reported bugs (hence I'm testing patches) I did NOT include Paulo's testcase patch for gcc.c-torture/execute/builtins/pr23484-chk.c: (Paulo please state effect on numbers for this) === gcc Summary === # of expected passes11785 # of unexpected failures59 # of unresolved testcases30 # of unsupported tests682 /cygdrive/e/awhconf/gcc/xgcc version 4.3.0 20080121 (experimental) (GCC) Unlike Paulo, it takes a while to run test on Cygwin as make spawns tasks. On the otherhand, I can download full GCC in maybe 30 mins and a new rev in minutes! Andy ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Here is my list: FAIL: gcc.c-torture/execute/20010122-1.c execution, -O0 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O1 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O2 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O3 -g FAIL: gcc.c-torture/execute/20010122-1.c execution, -Os FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O2 FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -g FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -Os FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O0 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O1 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O2 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -g FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -Os FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O0 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O1 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O2 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O3 -g FAIL: gcc.c-torture/execute/ffs-1.c compilation, -Os FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O0 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O1 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O2 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -g FAIL: gcc.c-torture/execute/ffs-2.c compilation, -Os FAIL: gcc.c-torture/execute/float-floor.c execution, -O0 FAIL: gcc.c-torture/execute/float-floor.c execution, -O1 FAIL: gcc.c-torture/execute/float-floor.c execution, -O2 FAIL: gcc.c-torture/execute/float-floor.c execution, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/float-floor.c execution, -O3 -g FAIL: gcc.c-torture/execute/float-floor.c execution, -Os FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O0 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O1 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O2 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -g FAIL: gcc.c-torture/execute/multi-ix.c compilation, -Os FAIL: gcc.c-torture/execute/pr17377.c execution, -O0 FAIL: gcc.c-torture/execute/pr17377.c execution, -O1 FAIL: gcc.c-torture/execute/pr17377.c execution, -O2 FAIL: gcc.c-torture/execute/pr17377.c execution, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/pr17377.c execution, -O3 -g FAIL: gcc.c-torture/execute/pr17377.c execution, -Os FAIL: gcc.c-torture/execute/pr22493-1.c execution, -O1 FAIL: gcc.c-torture/execute/pr22493-1.c execution, -O2 FAIL: gcc.c-torture/execute/pr22493-1.c execution, -Os FAIL: gcc.c-torture/execute/pr27364.c execution, -O1 FAIL: gcc.c-torture/execute/pr27364.c execution, -O2 FAIL: gcc.c-torture/execute/pr27364.c execution, -Os ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Delta [was: RE: [avr-gcc-list] More results from the testsuite with avrtest]
-Original Message- From: John Regehr [mailto:[EMAIL PROTECTED] Sent: Monday, January 21, 2008 8:58 PM To: Weddington, Eric Cc: Paulo Marques; Andrew Hutchinson; avr-gcc-list@nongnu.org; Andy Hutchinson Subject: RE: [avr-gcc-list] More results from the testsuite with avrtest Also this Delta implementation is very useful for minimizing an offending test program (either before or after narrowing settling on a version of gcc): http://delta.tigris.org/ I just used delta on WinAVR bug #1860717. It worked like a charm, even using Cygwin. :-) Thanks for pointing out this tool, John! Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Andrew Hutchinson [EMAIL PROTECTED]: Here is my list: [...] FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O0 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O1 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O2 FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O3 -g FAIL: gcc.c-torture/execute/multi-ix.c compilation, -Os These seem to be the bad guys. This test only fails for me with -O0, but passes on all other compiler options. I think the problem is that this test requires the increase in stack size in atmega128-exp.sim file, i.e., you have to replace set_board_info gcc,stack_size 400 with set_board_info gcc,stack_size 2048 or it won't have enough stack to run the test. Don't forget to also move bss to external memory with: set_board_info ldflags /home/pmarques/dejagnuboards/exit.c -Wl,-u,vfprintf -lprintf_flt -Wl,-Tbss=0x802000,--defsym=__heap_end=0x80 or some other tests might fail too. -- Paulo Marques This message was sent using IMP, the Internet Messaging Program. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
That explains it! ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
The bug with PR27364 testcase appears recent.
It fails with my 4.3 experimental copy 13/12/2007
Ok with avr-gcc (GCC) 4.2.2 (WinAVR 20071221)
reduced testcase is:
long f2(long number_of_digits_to_use)
{
return ( number_of_digits_to_use * 11L ) ;
}
___
AVR-GCC-list mailing list
AVR-GCC-list@nongnu.org
http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Paulo, please report bug with PR27364 testcase. I found that problem is somewhere in combine phase. Here it is trying to stick together pairs of instruction to make another. It fails but that is where load of constant dissappears. So if you post bug, I can add further info. Andy PS running your simulator now. Found bug in AVRORA - dunno where but it failed some testcases. Your's is same speed. Sadly Cygwin does not like spawn. Hopefully, our numbers will match! ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Andrew Hutchinson [EMAIL PROTECTED]: Here are my test results using Paulo simulator. Seems I have more tests! I'm now running from SVN, so I'm always testing the latest version, although my initial report was for gcc 4.2.2. One thing I noticed is that almost everyday there is a new test added to the testsuite. So, if we're running a svn version from a month ago, many tests may have entered, or may have been marked as unsuported in the meantime. I may have less failures as I did fix a bug I found while sorting out test patterns. The rest look the same as Paulo found. Ok, to try to synchronize with you, so that our tests are consistent, I'm running a test against svn revision 131704 (from today). Changes I've made so far to solve some of the failures: - created a sys/types.h with: #include inttypes.h #include stdint.h - changed ldflags definition in atmega128-sim.exp to: set_board_info ldflags /home/pmarques/dejagnuboards/exit.c -Wl,-u,vfprintf -lprintf_flt -Wl,-Tbss=0x802000,--defsym=__heap_end=0x80 the -Wl,-Tbss=0x802000,--defsym=__heap_end=0x80 part points the bss to external RAM, so that we have 4kb of just data + stack and 56k of BSS. This allows some of the tests that need a little more memory to run. - changed the stack size definition to half the internal RAM: set_board_info gcc,stack_size 2048 - already applied this patch to gcc.c-torture/execute/builtins/pr23484-chk.c: --- gcc/testsuite/gcc.c-torture/execute/builtins/pr23484-chk.c (revision 131704) +++ gcc/testsuite/gcc.c-torture/execute/builtins/pr23484-chk.c (working copy) @@ -41,8 +41,8 @@ abort (); memset (buf, 'L', sizeof (buf)); - if (snprintf (buf, l1 ? sizeof (buf) : 4, %d, l1 + 65536) != 5 - || memcmp (buf, 655\0, 8)) + if (snprintf (buf, l1 ? sizeof (buf) : 4, %d, l1 + 32760) != 5 + || memcmp (buf, 327\0, 8)) abort (); if (chk_calls) The result of running 'make -k check RUNTESTFLAGS=--target_board=atmega128-sim --all execute.exp' under these conditions is: === gcc Summary === # of expected passes11799 # of unexpected failures52 # of unresolved testcases 23 # of unsupported tests 682 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) The tests that fail for me are: 20010122-1.c execution, -O0 20010122-1.c execution, -O1 20010122-1.c execution, -O2 20010122-1.c execution, -O3 -g 20010122-1.c execution, -Os built-in-setjmp.c execution, -O2 built-in-setjmp.c execution, -O3 -fomit-frame-pointer built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-loops built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions built-in-setjmp.c execution, -O3 -g built-in-setjmp.c execution, -Os builtin-bitops-1.c compilation, -O0 builtin-bitops-1.c compilation, -O1 builtin-bitops-1.c compilation, -O2 builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-loops builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions builtin-bitops-1.c compilation, -O3 -g builtin-bitops-1.c compilation, -Os ffs-1.c compilation, -O0 ffs-1.c compilation, -O1 ffs-1.c compilation, -O2 ffs-1.c compilation, -O3 -fomit-frame-pointer ffs-1.c compilation, -O3 -g ffs-1.c compilation, -Os ffs-2.c compilation, -O0 ffs-2.c compilation, -O1 ffs-2.c compilation, -O2 ffs-2.c compilation, -O3 -fomit-frame-pointer ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-loops ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions ffs-2.c compilation, -O3 -g ffs-2.c compilation, -Os float-floor.c execution, -O0 float-floor.c execution, -O1 float-floor.c execution, -O2 float-floor.c execution, -O3 -fomit-frame-pointer float-floor.c execution, -O3 -g float-floor.c execution, -Os multi-ix.c compilation, -O0 pr17377.c execution, -O0 pr17377.c execution, -O1 pr17377.c execution, -O2 pr17377.c execution, -O3 -fomit-frame-pointer pr17377.c execution, -O3 -g pr17377.c execution, -Os pr22493-1.c execution, -O1 pr22493-1.c execution, -O2 pr22493-1.c execution, -Os pr27364.c execution, -O1 pr27364.c execution, -O2 pr27364.c execution, -Os Unresolved are all missing float function - or mmix? where data is to large (causes 8 unresolved) If you mean the missing __clzhi2, __ctzhi2, etc., these are bit operations that are missing from libgcc. Some of them might be used by the floating point emulation, though. Importantly, I looks like my other compiler patches work too! Great :) === gcc Summary === # of expected passes11663 # of unexpected failures59 # of unresolved testcases 30 # of unsupported tests 676 These numbers do look familiar, so there aren't probably much differences between our results. I've reported the
Re: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Andrew Hutchinson [EMAIL PROTECTED]: The bug with PR27364 testcase appears recent. It fails with my 4.3 experimental copy 13/12/2007 Ok with avr-gcc (GCC) 4.2.2 (WinAVR 20071221) This got me thinking: it would be really nice if we could setup an automated bisection test. We could start with something like: - known good revision: 117923 (gcc4.2 branch) - known bad revision: 131704 This gives us 13781 revisions to test. By bisecting, we should be able to lock in on the offending patch in less than 14 attempts. A script that automated this bisection process (checkout middle revision, test it, mark good/bad) would be great to find what exectly caused specific regressions. I may give it a try tomorrow, -- Paulo Marques This message was sent using IMP, the Internet Messaging Program. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Paulo Marques Sent: Monday, January 21, 2008 5:15 PM To: Andrew Hutchinson Cc: avr-gcc-list@nongnu.org; Andy Hutchinson Subject: Re: [avr-gcc-list] More results from the testsuite with avrtest Quoting Andrew Hutchinson [EMAIL PROTECTED]: The bug with PR27364 testcase appears recent. It fails with my 4.3 experimental copy 13/12/2007 Ok with avr-gcc (GCC) 4.2.2 (WinAVR 20071221) This got me thinking: it would be really nice if we could setup an automated bisection test. We could start with something like: - known good revision: 117923 (gcc4.2 branch) - known bad revision: 131704 This gives us 13781 revisions to test. By bisecting, we should be able to lock in on the offending patch in less than 14 attempts. A script that automated this bisection process (checkout middle revision, test it, mark good/bad) would be great to find what exectly caused specific regressions. I may give it a try tomorrow, Please see the GCC project about this. IIRC, they already have a script that does a binary search on failing test cases. IIRC, it is in the contrib subdirectory. Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Paulo Marques Sent: Monday, January 21, 2008 5:14 PM To: Andrew Hutchinson Cc: avr-gcc-list@nongnu.org Subject: Re: [avr-gcc-list] More results from the testsuite with avrtest One thing I noticed is that almost everyday there is a new test added to the testsuite. So, if we're running a svn version from a month ago, many tests may have entered, or may have been marked as unsuported in the meantime. I may have less failures as I did fix a bug I found while sorting out test patterns. The rest look the same as Paulo found. Ok, to try to synchronize with you, so that our tests are consistent, I'm running a test against svn revision 131704 (from today). Changes I've made so far to solve some of the failures: - created a sys/types.h with: #include inttypes.h #include stdint.h Note that adding inttypes.h *only* should be sufficient. inttypes.h automatically includes stdint.h. The result of running 'make -k check RUNTESTFLAGS=--target_board=atmega128-sim --all execute.exp' under these conditions is: === gcc Summary === # of expected passes11799 # of unexpected failures52 # of unresolved testcases 23 # of unsupported tests 682 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) Paulo, Andy, I owe you guys a beer! :-) These numbers are fantastic! Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
- created a sys/types.h with: #include inttypes.h #include stdint.h where do I put new include file? (directory relative to prefix root) I have updated my GCC to same rev as you. Ok with ldflags changes So hopefully I can re-run in sync with you. Andy ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Weddington, Eric [EMAIL PROTECTED]: [...] Ok, to try to synchronize with you, so that our tests are consistent, I'm running a test against svn revision 131704 (from today). Changes I've made so far to solve some of the failures: - created a sys/types.h with: #include inttypes.h #include stdint.h Note that adding inttypes.h *only* should be sufficient. inttypes.h automatically includes stdint.h. I didn't really look into it. I'll probably look closer to the builtins.exp part of the testsuite to see why is sys/types.h needed. Maybe it isn't really needed at all and we can just fix it there. Or maybe it is needed and this quick hack isn't really providing what the testsuite is expecting. The result of running 'make -k check RUNTESTFLAGS=--target_board=atmega128-sim --all execute.exp' under these conditions is: === gcc Summary === # of expected passes11799 # of unexpected failures52 # of unresolved testcases 23 # of unsupported tests 682 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) Paulo, Andy, I owe you guys a beer! :-) These numbers are fantastic! Thanks, but please bare in mind that these are the results for execute.exp alone. My latest result for the full testsuite is still: === gcc Summary === # of expected passes42683 # of unexpected failures574 # of unexpected successes 5 # of expected failures 89 # of unresolved testcases 248 # of untested testcases 64 # of unsupported tests 1250 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) If we consider an average of 5 tests failed per source file (with different compilation options), we are still ~128 problems away from a total cleanup. That doesn't mean you can skip that beer, though. After all, a promise is a promise ;) -- Paulo Marques This message was sent using IMP, the Internet Messaging Program. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: Paulo Marques [mailto:[EMAIL PROTECTED] Sent: Monday, January 21, 2008 6:14 PM To: Weddington, Eric Cc: Andrew Hutchinson; avr-gcc-list@nongnu.org Subject: RE: [avr-gcc-list] More results from the testsuite with avrtest Thanks, but please bare in mind that these are the results for execute.exp alone. My latest result for the full testsuite is still: === gcc Summary === # of expected passes42683 # of unexpected failures574 # of unexpected successes 5 # of expected failures 89 # of unresolved testcases 248 # of untested testcases 64 # of unsupported tests 1250 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) If we consider an average of 5 tests failed per source file (with different compilation options), we are still ~128 problems away from a total cleanup. This is still better. The last full testsuite run, on HEAD (future 4.3.0) was sent to the gcc-testresults list on October 26, 2007, by Mike Stein: http://gcc.gnu.org/ml/gcc-testresults/2007-10/msg01205.html Which shows: === gcc Summary === # of expected passes41563 # of unexpected failures715 # of unexpected successes 1 # of expected failures 95 # of unresolved testcases 287 # of untested testcases 65 # of unsupported tests 1200 /home/mstein/sim/avr/build/gcc/xgcc version 4.3.0 20071025 (experimental) [trunk revision 129634] (GCC) That doesn't mean you can skip that beer, though. After all, a promise is a promise ;) No worries. ;-D Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Ok my latest build is squawking - I assume because my binutils is out of step. ../../../../gcc/libgcc/../gcc/config/avr/libgcc.S:283: Error: illegal opcode mov w for mcu avr3 To make this painless, can someone please point me at correct sources for binutils - and assocated patch set (one that works with sources even :-P ) Andy Paulo Marques wrote: Quoting Weddington, Eric [EMAIL PROTECTED]: [...] Ok, to try to synchronize with you, so that our tests are consistent, I'm running a test against svn revision 131704 (from today). Changes I've made so far to solve some of the failures: - created a sys/types.h with: #include inttypes.h #include stdint.h Note that adding inttypes.h *only* should be sufficient. inttypes.h automatically includes stdint.h. I didn't really look into it. I'll probably look closer to the builtins.exp part of the testsuite to see why is sys/types.h needed. Maybe it isn't really needed at all and we can just fix it there. Or maybe it is needed and this quick hack isn't really providing what the testsuite is expecting. The result of running 'make -k check RUNTESTFLAGS=--target_board=atmega128-sim --all execute.exp' under these conditions is: === gcc Summary === # of expected passes11799 # of unexpected failures52 # of unresolved testcases 23 # of unsupported tests 682 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) Paulo, Andy, I owe you guys a beer! :-) These numbers are fantastic! Thanks, but please bare in mind that these are the results for execute.exp alone. My latest result for the full testsuite is still: === gcc Summary === # of expected passes42683 # of unexpected failures574 # of unexpected successes 5 # of expected failures 89 # of unresolved testcases 248 # of untested testcases 64 # of unsupported tests 1250 /home/pmarques/Desktop/gcc-4.2.2/svn/obj/gcc/xgcc version 4.3.0 20080119 (experimental) (GCC) If we consider an average of 5 tests failed per source file (with different compilation options), we are still ~128 problems away from a total cleanup. That doesn't mean you can skip that beer, though. After all, a promise is a promise ;) ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Andrew Hutchinson Sent: Monday, January 21, 2008 6:56 PM To: Paulo Marques; avr-gcc-list@nongnu.org Subject: Re: [avr-gcc-list] More results from the testsuite with avrtest Ok my latest build is squawking - I assume because my binutils is out of step. ../../../../gcc/libgcc/../gcc/config/avr/libgcc.S:283: Error: illegal opcode mov w for mcu avr3 To make this painless, can someone please point me at correct sources for binutils - and assocated patch set (one that works with sources even :-P ) Binutils 2.18. Patch set from WinAVR project's CVS on SourceForge (under patches module): http://winavr.cvs.sourceforge.net/winavr/ Note the README file under that module. Look under binutils/2.18 subdirectory. The patch filenames are numbered at the beginning, in categories (README file explains this). Patch in that order. You don't have to do the 0x patches (WinAVR specific), or the 1x patches (MinGW host specific, unless you're buildling for that host). Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
Re: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Andrew Hutchinson [EMAIL PROTECTED]: - created a sys/types.h with: #include inttypes.h #include stdint.h where do I put new include file? (directory relative to prefix root) In Linux, the default dir is /usr/local/avr/include so the full path for the file becomes /usr/local/avr/include/sys/types.h I have updated my GCC to same rev as you. That was fast. Ok with ldflags changes So hopefully I can re-run in sync with you. Great. I'm really curious about the results, -- Paulo Marques This message was sent using IMP, the Internet Messaging Program. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Weddington, Eric [EMAIL PROTECTED]: Ok my latest build is squawking - I assume because my binutils is out of step. ../../../../gcc/libgcc/../gcc/config/avr/libgcc.S:283: Error: illegal opcode mov w for mcu avr3 To make this painless, can someone please point me at correct sources for binutils - and assocated patch set (one that works with sources even :-P ) Binutils 2.18. Patch set from WinAVR project's CVS on SourceForge (under patches module): http://winavr.cvs.sourceforge.net/winavr/ Note the README file under that module. Look under binutils/2.18 subdirectory. The patch filenames are numbered at the beginning, in categories (README file explains this). Patch in that order. You don't have to do the 0x patches (WinAVR specific), or the 1x patches (MinGW host specific, unless you're buildling for that host). FWIW, I've been running my tests with binutils just plain 2.18. After reading this I decided to try and apply the patches, but at least for the execute.exp test it didn't make a difference in the results. -- Paulo Marques This message was sent using IMP, the Internet Messaging Program. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
Please see the GCC project about this. IIRC, they already have a script that does a binary search on failing test cases. IIRC, it is in the contrib subdirectory. Also this Delta implementation is very useful for minimizing an offending test program (either before or after narrowing settling on a version of gcc): http://delta.tigris.org/ John ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: John Regehr [mailto:[EMAIL PROTECTED] Sent: Monday, January 21, 2008 8:58 PM To: Weddington, Eric Cc: Paulo Marques; Andrew Hutchinson; avr-gcc-list@nongnu.org; Andy Hutchinson Subject: RE: [avr-gcc-list] More results from the testsuite with avrtest Please see the GCC project about this. IIRC, they already have a script that does a binary search on failing test cases. IIRC, it is in the contrib subdirectory. Also this Delta implementation is very useful for minimizing an offending test program (either before or after narrowing settling on a version of gcc): http://delta.tigris.org/ Very interesting tool! Thanks for the link! Eric W. ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Paulo Marques Sent: Saturday, January 19, 2008 11:27 AM To: avr-gcc-list@nongnu.org Subject: [avr-gcc-list] More results from the testsuite with avrtest Hi, all I've been running the testsuite from SVN with avrtest to track down the latest bugs. I've finally track down every failure in execute.exp, and I'm preparing to move on to the full testsuite (I hope). So, here's a list of all the problems so far: FAIL: gcc.c-torture/execute/20010122-1.c execution, -O0 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O1 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O2 FAIL: gcc.c-torture/execute/20010122-1.c execution, -O3 -g FAIL: gcc.c-torture/execute/20010122-1.c execution, -Os There is something fishy going on with __builtin_return_address. This test in particular only fails when it is compiled with -Wl,-u,vfprintf -lprintf_flt. This probably is just an indirect effect of some code reordering due to the increased image size, or something. Still needs more investigation. This one is already recorded as GCC bug #21078, Bjoern Haase (CCed): http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21078 Eric Weddington ___ AVR-GCC-list mailing list AVR-GCC-list@nongnu.org http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
RE: [avr-gcc-list] More results from the testsuite with avrtest
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Paulo Marques Sent: Saturday, January 19, 2008 11:27 AM To: avr-gcc-list@nongnu.org Subject: [avr-gcc-list] More results from the testsuite with avrtest Hi, all I've been running the testsuite from SVN with avrtest to track down the latest bugs. I've finally track down every failure in execute.exp, and I'm preparing to move on to the full testsuite (I hope). So, here's a list of all the problems so far: snip FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O2 FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O3 -g FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -Os This is an actual bug, that only appears with O2, O3 or Os. The __builtin_setjmp stores Y+1 in the setjmp buffer. With -O0 the first instruction after the jmp does a sbiw r28, 1 that restores the original value, but for some reason, with higher optimization levels, this instruction is optimized away, leaving R28 pointing to the wrong address by one. The __builtin_setjmp code: if (__builtin_setjmp (buf)) 16a: ce 01 movwr24, r28 16c: 01 96 adiwr24, 0x01 ; 1 Notice the increment here, before storing R28 16e: 90 93 07 01 sts 0x0107, r25 172: 80 93 06 01 sts 0x0106, r24 176: 8f ee ldi r24, 0xEF ; 239 178: 90 e0 ldi r25, 0x00 ; 0 17a: 90 93 09 01 sts 0x0109, r25 17e: 80 93 08 01 sts 0x0108, r24 182: ed b7 in r30, 0x3d ; 61 184: fe b7 in r31, 0x3e ; 62 186: f0 93 0b 01 sts 0x010B, r31 18a: e0 93 0a 01 sts 0x010A, r30 The __builtin_longjmp code: __builtin_longjmp (buf, 1); 10c: e0 91 08 01 lds r30, 0x0108 110: f0 91 09 01 lds r31, 0x0109 114: c0 91 06 01 lds r28, 0x0106 118: d0 91 07 01 lds r29, 0x0107 11c: 80 91 0a 01 lds r24, 0x010A 120: 90 91 0b 01 lds r25, 0x010B no decrement, R28 is used as is. FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O0 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O1 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O2 FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O3 -g FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -Os This test fails compilation with these errors: builtin-bitops-1.c:(.text+0x672): undefined reference to `__ffshi2' builtin-bitops-1.c:(.text+0x6b4): undefined reference to `__popcounthi2' builtin-bitops-1.c:(.text+0x6ee): undefined reference to `__parityhi2' builtin-bitops-1.c:(.text+0x183e): undefined reference to `__clzhi2' builtin-bitops-1.c:(.text+0x1872): undefined reference to `__ctzhi2' Maybe these should be defined in avr-libc? Is there something wrong with my setup? No. AFAIK, these are not defined for the AVR. All of the above look like real bugs. I don't know of any bug reports for these (yet). FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O0 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O1 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O2 FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/ffs-1.c compilation, -O3 -g FAIL: gcc.c-torture/execute/ffs-1.c compilation, -Os This one just needs: ffs-1.c:(.text+0x8): undefined reference to `__ffshi2' FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O0 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O1 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O2 FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-loops FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions FAIL: gcc.c-torture/execute/ffs-2.c compilation, -O3 -g FAIL: gcc.c-torture/execute/ffs-2.c compilation, -Os And again: ffs-2.c:(.text+0x8): undefined reference to `__ffshi2' ffs-2.c:(.text+0x20): undefined reference to `__ffshi2' ffs-2.c:(.text+0x38): undefined reference to `__ffshi2' ffs-2.c:(.text+0x50): undefined reference to `__ffshi2' Again,
RE: [avr-gcc-list] More results from the testsuite with avrtest
Quoting Weddington, Eric [EMAIL PROTECTED]:
[...]
FAIL: gcc.c-torture/execute/built-in-setjmp.c execution, -O2
Now reported as bug #34879:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=34879
FAIL: gcc.c-torture/execute/builtin-bitops-1.c compilation, -O0
[...]
The undefined reference to __ffshi2 had already been reported:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=34210
The other errors seemed similar enough to be added to the same report,
so I just added a new comment.
more identical problems snipped
FAIL: gcc.c-torture/execute/float-floor.c execution, -O0
[...]
This test seems to only work if we have a 8 byte double type. So,
this should probably be unsupported for avr.
Agreed. Although at some point it would be nice to have 8 byte double
types for the AVR.
Yes, it would be nice, but in the meantime, the test should be fixed so
that we can run the test suite with no failures:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=34880
FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O0
FAIL: gcc.c-torture/execute/multi-ix.c compilation, -O1
[...]
If I change the STACK_SIZE to 1000, the test runs successfully. Since
the atmega128 has 4Kb of RAM, maybe we should increase our
requirements a little.
Agreed.
From what I remember, there are some other test in the test suite that
fail on the AVR because the AVR does not have enough RAM to support
those tests. In those cases, those tests should be disabled for the AVR.
For avrtest, I could set the RAM to 64k and pretend I had external RAM,
by giving the proper parameters to the compiler / linker. This should
allow us to run more of the RAM demanding tests. I'll give it a try.
FAIL: gcc.c-torture/execute/pr17377.c execution, -O0
[...]
This also uses __builtin_return_address. Still tracking down this one.
See AVR GCC bug #21080
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21080
It references __builtin_return_address. A comment in that bug also
references bug #21078:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21078
Which I mentioned earlier.
So it's possible that these could all be related.
Ok. I'll wait for those bugs to be closed and retry later.
FAIL: gcc.c-torture/execute/pr22493-1.c execution, -O1
FAIL: gcc.c-torture/execute/pr22493-1.c execution, -O2
FAIL: gcc.c-torture/execute/pr22493-1.c execution, -Os
This is an actual bug. This function:
void f(int i)
{
if (i0)
abort();
i = -i;
if (i0)
return;
abort ();
}
is compiled to:
void f(int i)
{
abort ();
}
because if (i = 0), then (-i = 0) must be true, right?
Unfortunately this is wrong for the corner case where i = INT_MIN,
because -i is also INT_MIN.
Then it probably needs a bug report.
After looking at it some more, I'm starting to wonder: i = -i is
actually a signed overflow for the case where i = INT_MIN. So, by the
standard, the result is undefined, and the compiler is free to optimize
this.
Now, gcc has a couple of flags to deal with this sort of optimizations:
-fwrapv and -fno-strict-overflow. They do produce different code, but
still assume that i = -i; if (i 0) is equivalent to if (i = 0).
So, is the testsuite that needs fixing or gcc?
FAIL: gcc.c-torture/execute/pr27364.c execution, -O1
FAIL: gcc.c-torture/execute/pr27364.c execution, -O2
FAIL: gcc.c-torture/execute/pr27364.c execution, -Os
This test assumes 32 bit integers.
Then the test itself needs to be fixed. There have been a number of
cases like this recently where the test would fail on another 16-bit
integer target, and the test was fixed so as to not make that
assumption.
Unfortunately, there is more to this test than meets the eye :(
Even after I changed it to:
int f(unsigned number_of_digits_to_use)
{
if (number_of_digits_to_use 1294)
return 0;
return ((number_of_digits_to_use * 3321928L) / 100L + 1) /16;
}
the test still fails with -O1, -O2 and -Os. With -O0 it produces
correct code, and with -O3, the f function is incorrect (as with -O2),
but the call is optimized away in the main function.
With -O2 we get this:
int f(unsigned number_of_digits_to_use)
{
if (number_of_digits_to_use 1294)
206:65 e0 ldi r22, 0x05 ; 5
208:8f 30 cpi r24, 0x0F ; 15
20a:96 07 cpc r25, r22
20c:c0 f4 brcc.+48; 0x23e f+0x38
return 0;
return ((number_of_digits_to_use * 3321928L) / 100L + 1) /16;
20e:bc 01 movwr22, r24
210:80 e0 ldi r24, 0x00 ; 0
212:90 e0 ldi r25, 0x00 ; 0
214:0e 94 6c 06 call0xcd8 ; 0xcd8 __mulsi3
[...]
It forgets to load r18:r19:r20:r21 with 3321928 before calling __mulsi3.
So maybe I should file 2 bug reports... :P
snipped bug reporting instructions
A VERY BIG THANK YOU! for doing all of this! This work will go a long
way towards improving the quality of the AVR toolchain!
Thank you for your support :)
--
Paulo Marques
