From: Alexis Lothoré <[email protected]> Hello, here is another batch of fixes to reduce noise in regression reports. Fixes are directly linked to main noise sources seen in 4.2_M3 regression report ([1]). - fix some existing selftests for resulttool - add more filters for ptests incorrectly logging failures. The root cause of those wrongly named test results remained to be fixed to stop saving broken test results, but this series fixes at least parsing for existing results - Stop logging "newly passing" tests in regression reports. Some real regressions are "hidden" in big chunks of newly passing tests: foo: FAIL -> PASS bar: FAIL -> PASS moo: FAIL -> PASS [...] xxx: PASS -> FAIL [...] yyy: FAIL -> PASS zzz: FAIL -> PASS
We are reaching a point where regression reports are small enough to get posted on pastebin, so by following [2] you can find a report sample generated with tooling patched with this series [1] https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt [2] https://pastebin.com/QgfLKhpx Alexis Lothoré (6): scripts/resulttool: call fixup_ptest_names in regression_common oeqa/selftest/resulttool: fix ptest filtering tests oeqa/selftest/resulttool: fix fake data used for testing scripts/resulttool: fix ptests results containing a non reproducible path oeqa/selftest/resulttool: add test for error propagation in test name filtering scripts/resulttool: do not count newly passing tests as regressions .../oeqa/selftest/cases/resulttooltests.py | 112 +++++++++++------- scripts/lib/resulttool/regression.py | 66 +++++++---- 2 files changed, 112 insertions(+), 66 deletions(-) -- 2.39.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#177847): https://lists.openembedded.org/g/openembedded-core/message/177847 Mute This Topic: https://lists.openembedded.org/mt/97296204/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
