On Tue, Feb 28, 2023 at 10:10 AM Alexis Lothoré via lists.openembedded.org <[email protected]> wrote:
> From: Alexis Lothoré <[email protected]> > > Hello, > here is another batch of fixes to reduce noise in regression reports. > Fixes are > directly linked to main noise sources seen in 4.2_M3 regression report > ([1]). > - fix some existing selftests for resulttool > - add more filters for ptests incorrectly logging failures. The root cause > of > those wrongly named test results remained to be fixed to stop saving > broken > test results, but this series fixes at least parsing for existing results > - Stop logging "newly passing" tests in regression reports. Some real > regressions are "hidden" in big chunks of newly passing tests: > foo: FAIL -> PASS > bar: FAIL -> PASS > moo: FAIL -> PASS > [...] > xxx: PASS -> FAIL > [...] > yyy: FAIL -> PASS > zzz: FAIL -> PASS > > We are reaching a point where regression reports are small enough to get > posted > on pastebin, so by following [2] you can find a report sample generated > with > tooling patched with this series > > [1] https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/t > <https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt> It seems a bit odd that all the regressions are changing from a valid state (PASS, SKIP…) -> None. Does this literally mean the only changes were dropped test cases? estresults/testresult-regressions-report.txt > <https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt> > [2] https://pastebin.com/QgfLKhpx > > Alexis Lothoré (6): > scripts/resulttool: call fixup_ptest_names in regression_common > oeqa/selftest/resulttool: fix ptest filtering tests > oeqa/selftest/resulttool: fix fake data used for testing > scripts/resulttool: fix ptests results containing a non reproducible > path > oeqa/selftest/resulttool: add test for error propagation in test name > filtering > scripts/resulttool: do not count newly passing tests as regressions > > .../oeqa/selftest/cases/resulttooltests.py | 112 +++++++++++------- > scripts/lib/resulttool/regression.py | 66 +++++++---- > 2 files changed, 112 insertions(+), 66 deletions(-) > > -- > 2.39.1 > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#177867): https://lists.openembedded.org/g/openembedded-core/message/177867 Mute This Topic: https://lists.openembedded.org/mt/97296204/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
