Re: Question about gcc test suite failures
On Wed, Feb 20, 2019 at 03:04:51PM +0100, Tomasz Kłoczko wrote:
> OK. Jakub so what kind of method you are using to recognise that something
> is wrong with gcc/gdb/binutils if none of those packages test suites seems
> may exit with non-zero exit code and some failures are perfectly OK?
> (Or maybe logic of the suite should be negated?)
> Do you have some sort of script/tool extracting results from build logs? or
> it is some other method?
> Syntax of the content /^==*TESTING/,/^==*TESTING END/p' looks a bit
> complicated ..
> How to trace those results in more or less automated way?
It is intentionally a manual process.
For koji, my current script is:
#!/bin/sh
vr=`echo $2 | sed 's,/,-,'`
mkdir -p ~/tool-test/$1/$vr
for arch in i686 x86_64 armv7hl ppc64 ppc64le aarch64 s390 s390x ppc; do
ar=`echo $arch | sed 's/armv7hl/armhfp/;s/i686/i386/'`
wget
http://kojipkgs.fedoraproject.org/packages/$1/$2/data/logs/$arch/build.log -O
~/tool-test/$1/$vr/$ar
if [ ! -s ~/tool-test/$1/$vr/$ar ]; then rm ~/tool-test/$1/$vr/$ar; continue;
fi
sed -n -i -e '/^==*TESTING/,/^==*TESTING END/p' ~/tool-test/$1/$vr/$ar
done
which populates the directories, and then I just
diff -uprN ~/tool-test/gcc/9.0.1-0.{3,4}.fc30
and use my own judgement of what is and what isn't a blocker and what needs
to be fixed even if it is not a blocker.
Jakub
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Question about gcc test suite failures
On Wed, 20 Feb 2019 at 14:51, Jakub Jelinek wrote: [..] > The aim is to avoid regressions, not zero FAIL rate, which e.g. for the > guality testcase is not really possible as the testing matrix is too large > for debug info coverage, -O levels x targets x ISA choices x GDB issues > and it is impossible to encode that into xfails. > Plus, the rpm builds test both normally and with additional > -fstack-protector-strong as that is what the whole distro does, while > normally the latter isn't included. So, some FAILs might be once with > normal options and once with -fstack-protector-strong, other tests only > FAIL > with the latter e.g. when testing asan and having UB in the test to test it > how asan handles it and -fstack-protector-strong may stand in a way of > that. > There are some flakey tests too (mostly in the go testsuite). > > So, I have a directory with years of testsuite results from koji/brew > builds, essentially build logs from all the architectures massaged through > sed -n -e '/^==*TESTING/,/^==*TESTING END/p' > and compare that regularly for new builds. > OK. Jakub so what kind of method you are using to recognise that something is wrong with gcc/gdb/binutils if none of those packages test suites seems may exit with non-zero exit code and some failures are perfectly OK? (Or maybe logic of the suite should be negated?) Do you have some sort of script/tool extracting results from build logs? or it is some other method? Syntax of the content /^==*TESTING/,/^==*TESTING END/p' looks a bit complicated .. How to trace those results in more or less automated way? kloczek -- Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Question about gcc test suite failures
On Wed, Feb 20, 2019 at 02:31:37PM +0100, Tomasz Kłoczko wrote: > Recently I've been trying to trace some gcc issue so I've downloaded my gcc > src.rpm to try to build my own package. > During review build log I found a lot of test suite failures. > Initially I've been thinking that something is wrong with my devel env so > I've peaked on official gcc build logs and I found that I have even less > such failures than number of such issues on rawhide builders. The aim is to avoid regressions, not zero FAIL rate, which e.g. for the guality testcase is not really possible as the testing matrix is too large for debug info coverage, -O levels x targets x ISA choices x GDB issues and it is impossible to encode that into xfails. Plus, the rpm builds test both normally and with additional -fstack-protector-strong as that is what the whole distro does, while normally the latter isn't included. So, some FAILs might be once with normal options and once with -fstack-protector-strong, other tests only FAIL with the latter e.g. when testing asan and having UB in the test to test it how asan handles it and -fstack-protector-strong may stand in a way of that. There are some flakey tests too (mostly in the go testsuite). So, I have a directory with years of testsuite results from koji/brew builds, essentially build logs from all the architectures massaged through sed -n -e '/^==*TESTING/,/^==*TESTING END/p' and compare that regularly for new builds. Jakub ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
