On 2/16/23 01:02, Richard Purdie wrote:
> On Tue, 2023-02-14 at 17:53 +0100, Alexis Lothoré via
> lists.openembedded.org wrote:
>> From: Alexis Lothoré <[email protected]>
>> * this serie prioritize retro-compatibility: if the base test is older (ie:
>> it
>> does not have the needed metadata), it will consider tests as "comparable"
>> * additionally to tests added in oeqa test cases, some "best effort" manual
>> testing has been done, with the following cases:
>> - run a basic test (e.g: `oeselftest -r tinfoils`), collect test result,
>> break
>> test, collect result, ensure tests are compared. Change oeselftest
>> parameters, ensure tests are not compared
>> - collect base and target tests results from 4.2_M2 regression report,
>> manually add new metadata to some tests, replay regression report, ensure
>> that regressions are kept or discarded depending on the metadata
>
> I think this is heading in the right direction. Firstly, can we put
> some kind of test script into OE-Core for making debugging/testing this
> easier?
>
> I'm wondering if we can take some of the code from qa_send_email and
> move it into OE-Core such that I could do something like:
>
> show-regression-report 4.2_M1 4.2_M2
>
> which would then resolve those two tags to commits, find the
> testresults repo, fetch the data depth1 then call resulttool regression
> on them.
That totally makes sense, it will make future change easier to test, thanks for
the suggestion. I will think about how I can rework/transfer some code to do
that. I feel that one thing could still be troublesome to do so: since there
will be linked modification in yocto-autobuilder-helper and openembedded, do you
know if yocto-autobuilder-helper is fetched on same branch family as all
repositories needed for build ? From a quick reading, I see that it is by
default on "master" in config.py in yocto-autobuilder2 repository, so I think it
is always on master, which will be an issue to solve ?
>
> I did that manually to experiment. I realised that if we do something
> like:
>
> if "MACHINE" in base_configuration and "MACHINE" in target_configuration:
> if base_configuration["MACHINE"] != target_configuration["MACHINE"]:
> print("Skipping")
> return False
>
> in metadata_matches() we can skip a lot of mismatched combinations even
> with the older test results.
Indeed, comparing MACHINE when present on both results can help too. I may have
skipped this because of confusion about our previous discussion on some tests
being "cross-compared" while other are "strictly" compared ! I will ensure that
most of scenarios allow comparing MACHINE, and if so, I will add it in
metadata_matches to be checked first (because cheap to test), and if it matches,
check OESELFTEST_METADATA too (more expensive)
> I think we also should be able to use some
> pattern matching to generate a dummy OESELFTEST_METADATA section for
> older data which doesn't have it. For example, the presence of meta_ide
> tests indicates one particular type of test. Combined with the MACHINE
> match, this should let us compare old and new data? That would mean
> metadata_matches() would need to see into the actual results too.
I thought about this possibility while starting to work on this (checking for
existing metadata to kind of generate/guess test configuration), but I have felt
that it would need too much "business logic" encoding in resulttool scripts,
which could easily break if we want to change some tests configuration. I mean,
if someone changes some parameters passed to oe-selftest in builders
configuration, he will very probably not remember to update some "guess" filters
in resulttool.
I will start working on new revision of this series based on your comments
Kind regards,
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#177278):
https://lists.openembedded.org/g/openembedded-core/message/177278
Mute This Topic: https://lists.openembedded.org/mt/96964070/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
