On 20/01/2026 12:47, Alice Carlotti via Gcc wrote: > On Tue, Jan 20, 2026 at 01:04:15PM +0100, Georg-Johann Lay via Gcc wrote: >> Since quite some time now I am observing differences in testsuite >> results, even in the case when the compiler is build on the same >> host from the same sources, and configured the same way. >> >> What I did: >> >> Host is cfarm421: x86_64 GNU/Linux, Debian 6.12.57-1 >> >> 1) Checked out same GCC sources (usually trunk) to >> >> $HOME/gnu/source/gcc-master >> $HOME/gnu/source/gcc-x >> >> 2) In $HOME/gnu/build/gcc-host configure + build + test: >> >> $ ../../source/gcc-master/configure --enable-languages=c,c++ >> --disable-bootstrap >> $ make -j44 >> $ (cd gcc; make -j44 -k check-gcc) >> >> 3) In $HOME/gnu/build/gcc-x-host configure + build + test: >> >> $ ../../source/gcc-x/configure --enable-languages=c,c++ --disable-bootstrap >> $ make -j44 >> $ (cd gcc; make -j44 -k check-gcc) >> >> 4) Then compare the test outcomes: >> >> $ $HOME/gnu/source/gcc-master/contrib/compare_tests >> ~/gnu/build/gcc-host/gcc/testsuite/ ~/gnu/build/gcc-x-host/gcc/testsuite/ >> >> compare_tests's output reads: >> >> # Comparing directories >> ## Dir1=/home/gjl/gnu/build/gcc-host/gcc/testsuite/: 1 sum files >> ## Dir2=/home/gjl/gnu//build/gcc-x-host/gcc/testsuite/: 1 sum files >> >> # Comparing 1 common sum files >> ## /bin/sh /home/gjl/gnu/source/gcc-master/contrib/compare_tests >> /tmp/gxx-sum1.4194012 /tmp/gxx-sum2.4194012 >> Non-unique test names: (Eeek!) >> 2 PASS: gcc: c-c++-common/diagnostic-format-sarif-file-Wbidi-chars.c >> -Wc++-compat scan-sarif-file "text": "unpaired UTF-8 bidirectional control >> characters detected" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<80>'" >> 6 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<bf>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<c0>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<c1>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<c2>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<e0>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<e0><80><bf>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<e0><9f><80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<e0><bf>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<ec><80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<ed><a0><80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<f0><80><80><80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<f0><8f><bf><bf>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<f4><90><80><80>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<f5>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character >> '<fd><bf><bf><bf>'" >> 2 PASS: gcc: >> c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c >> -Wc++-compat scan-sarif-file "text": "invalid UTF-8 character '<ff>'" >> 3 PASS: gcc: c-c++-common/goacc/combined-reduction.c >> scan-tree-dump-times gimple "acc loop reduction.+:v1. private.i." 1 >> 3 PASS: gcc: c-c++-common/gomp/append-args-6.c scan-tree-dump-times >> gimple "D.[0-9]+ = __builtin_omp_get_mapped_ptr \\(cp2, 4\\);" 1 >> 2 PASS: gcc: c-c++-common/gomp/requires-5.c scan-tree-dump original >> "#pragma omp atomic release" >> 3 PASS: gcc: c-c++-common/pr57371-1.c -Wc++-compat scan-tree-dump >> original "return (<retval> = )?x != 0" >> 2 PASS: gcc: gcc.dg/attr-noipa.c scan-tree-dump optimized "fn20 \\(\\)" >> 2 PASS: gcc: gcc.dg/format/diagnostic-ranges.c >> diagnostic-ranges-html.py::test_annotation >> 2 PASS: gcc: gcc.dg/format/diagnostic-ranges.c >> diagnostic-ranges-html.py::test_messag >> 3 PASS: gcc: gcc.dg/lto/20111207-1 c_lto_20111207-1_0.o assemble, >> -fcommon >> 3 PASS: gcc: gcc.dg/lto/20111207-1 >> c_lto_20111207-1_0.o-c_lto_20111207-1_3.o execute -fcommon >> 3 PASS: gcc: gcc.dg/lto/20111207-1 >> c_lto_20111207-1_0.o-c_lto_20111207-1_3.o link, -fcommon >> 3 PASS: gcc: gcc.dg/lto/20111207-1 c_lto_20111207-1_1.o assemble, >> -fcommon >> 3 PASS: gcc: gcc.dg/lto/20111207-1 c_lto_20111207-1_2.o assemble, >> -fcommon >> 3 PASS: gcc: gcc.dg/lto/20111207-1 c_lto_20111207-1_3.o assemble, >> -fcommon >> 2 PASS: gcc: gcc.dg/sarif-output/multiple-outputs.c (test .sarif >> output for UTF-8-encoded parseable JSON) >> 2 UNSUPPORTED: gcc: gcc.dg/sarif-output/multiple-outputs.c >> verify-sarif-file: check-jsonschema is missing >> 2 PASS: gcc: gcc.dg/vect/vect-fmin-1.c -flto -ffat-lto-objects >> scan-tree-dump vect "Detected reduction" >> 2 PASS: gcc: gcc.dg/vect/vect-fmin-1.c scan-tree-dump vect "Detected >> reduction" >> 2 PASS: gcc: gcc.dg/vect/vect-reduc-dot-s16a.c -flto -ffat-lto-objects >> scan-tree-dump-times vect "vectorized 1 loops" 1 >> 2 PASS: gcc: gcc.dg/vect/vect-reduc-dot-s16a.c scan-tree-dump-times >> vect "vectorized 1 loops" 1 >> >> ## Differences found >> # 1 differences in 1 common sum files found >> >> >> I also see similar diffs for other targets. >> >> Can someone give a hint what I am doing wrong and how to fix that? > > You're not doing anything wrong, and there are no real differences. It's > just that the test script is misleading. > > All of the issues reported by the script are relating to duplicate test names, > which can be a problem because it leads to ambiguity about which test sum file > line refers to. I recall that Richard Earnshaw eliminated a lot of duplicate > names last year, but obviously some still remain. > > The line "1 differences in 1 common sum files found" doesn't actually count > differences. The second number is the number of sum files that are present in > both sets of test results, while the first number counts how many of these > files the script detected an issue in. As you have encountered here, not all > such issues are actual "differences". > > Alice > >> >> Johann >> >>
Yes, these are bugs in the testsuite, not bugs in the compiler itself. All tests should have unique names so that we can correctly check and detect if a test transitions state. Non unique names makes this nearly impossible. I'd hoped that when the change to the analysis script went in that most of the issues it reported would be fixed pretty quickly (as Alice said, I fixed a lot of them myself), but there are a couple of sub-frameworks in the compiler testsuite that are particularly bad in this regard. R.
