Hello Tim, On 3/1/23 07:24, Tim Orling wrote: > On Tue, Feb 28, 2023 at 10:10 AM Alexis Lothoré via lists.openembedded.org > <http://lists.openembedded.org> > <[email protected] > <mailto:[email protected]>> wrote: > > From: Alexis Lothoré <[email protected] > <mailto:[email protected]>> > > Hello, > here is another batch of fixes to reduce noise in regression reports. > Fixes are > directly linked to main noise sources seen in 4.2_M3 regression report > ([1]). > - fix some existing selftests for resulttool > - add more filters for ptests incorrectly logging failures. The root > cause of > those wrongly named test results remained to be fixed to stop saving > broken > test results, but this series fixes at least parsing for existing > results > - Stop logging "newly passing" tests in regression reports. Some real > regressions are "hidden" in big chunks of newly passing tests: > foo: FAIL -> PASS > bar: FAIL -> PASS > moo: FAIL -> PASS > [...] > xxx: PASS -> FAIL > [...] > yyy: FAIL -> PASS > zzz: FAIL -> PASS > > We are reaching a point where regression reports are small enough to get > posted > on pastebin, so by following [2] you can find a report sample generated > with > tooling patched with this series > > [1] https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/t > > <https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt> > > > It seems a bit odd that all the regressions are changing from a valid state > (PASS, SKIP…) -> None. Does this literally mean the only changes were dropped > test cases?
For most of "XXX -> None" transitions in the 4.2_M3.rc1 regression report, tests were not dropped between base and target tests, they are present in both, but the test names saved and stored in git are incorrect and bear a "non reproducible" part, which makes the tooling raise many of those wrong transitions. Here is an example: ptestresult.binutils-ld.in testcase /home/pokybuild/yocto-worker/qemux86/build/build-st-15167/tmp/work/core2-32-poky-linux/binutils-cross-testsuite/2.40-r0/git/ld/testsuite/ld-ctf/ctf.exp: ERROR -> None This binutils-ld test result is present in both base and target results, but the test name is very likely broken: the error has been captured as part of the test name, and worse than than, it contains multiple parts that change between executions (possibly "core2-32-poky-linux" and "qemux86", but especially "build-st-15167") because of path embedded in the error log. So when running "resulttool regression-git", the tool does not find in target the test it has found in base, which raises a "XXX -> None". Obviously the main issue has to be fixed in all runners generating those errors (so far, we have seen this kind of issues with ptests for binutils, curl, dbus, toolchains, glibc, etc), but since we want to be able to work with current tests results history, we must make the tools able to circumvent those issues. Regards, -- Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#177871): https://lists.openembedded.org/g/openembedded-core/message/177871 Mute This Topic: https://lists.openembedded.org/mt/97296204/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
