On Thu, 2023-02-16 at 09:56 +0100, Alexis Lothoré wrote:
> On 2/16/23 01:02, Richard Purdie wrote:
> > On Tue, 2023-02-14 at 17:53 +0100, Alexis Lothoré via
> > lists.openembedded.org wrote:
> > > From: Alexis Lothoré <[email protected]>
> > > * this serie prioritize retro-compatibility: if the base test is older
> > > (ie: it
> > > does not have the needed metadata), it will consider tests as "comparable"
> > > * additionally to tests added in oeqa test cases, some "best effort"
> > > manual
> > > testing has been done, with the following cases:
> > > - run a basic test (e.g: `oeselftest -r tinfoils`), collect test
> > > result, break
> > > test, collect result, ensure tests are compared. Change oeselftest
> > > parameters, ensure tests are not compared
> > > - collect base and target tests results from 4.2_M2 regression report,
> > > manually add new metadata to some tests, replay regression report,
> > > ensure
> > > that regressions are kept or discarded depending on the metadata
> >
> > I think this is heading in the right direction. Firstly, can we put
> > some kind of test script into OE-Core for making debugging/testing this
> > easier?
> >
> > I'm wondering if we can take some of the code from qa_send_email and
> > move it into OE-Core such that I could do something like:
> >
> > show-regression-report 4.2_M1 4.2_M2
I was thinking "yocto-testresult-query regression-report" might be a
better name/command, then it can default to using the yocto-testresults
repo and resolving yocto tags. You'll also likely want to specify a
workdir for the operation but those are implementation details :)
> >
> > which would then resolve those two tags to commits, find the
> > testresults repo, fetch the data depth1 then call resulttool regression
> > on them.
>
> That totally makes sense, it will make future change easier to test, thanks
> for
> the suggestion. I will think about how I can rework/transfer some code to do
> that. I feel that one thing could still be troublesome to do so: since there
> will be linked modification in yocto-autobuilder-helper and openembedded, do
> you
> know if yocto-autobuilder-helper is fetched on same branch family as all
> repositories needed for build ? From a quick reading, I see that it is by
> default on "master" in config.py in yocto-autobuilder2 repository, so I think
> it
> is always on master, which will be an issue to solve ?
Unlike yocto-autobuilder2, there is a branch of yocto-autobuilder-
helper per release. We can therefore move code from there to OE-Core on
the master branch and we'll be ok. It will be a small headache for SWAT
whilst we test it but Alexandre is on cc and will cope :).
> >
> > I did that manually to experiment. I realised that if we do something
> > like:
> >
> > if "MACHINE" in base_configuration and "MACHINE" in
> > target_configuration:
> > if base_configuration["MACHINE"] != target_configuration["MACHINE"]:
> > print("Skipping")
> > return False
> >
> > in metadata_matches() we can skip a lot of mismatched combinations even
> > with the older test results.
>
> Indeed, comparing MACHINE when present on both results can help too. I may
> have
> skipped this because of confusion about our previous discussion on some tests
> being "cross-compared" while other are "strictly" compared ! I will ensure
> that
> most of scenarios allow comparing MACHINE, and if so, I will add it in
> metadata_matches to be checked first (because cheap to test), and if it
> matches,
> check OESELFTEST_METADATA too (more expensive)
MACHINE should always match, that is fine. What doesn't need to match
are the distros and so on.
> > I think we also should be able to use some
> > pattern matching to generate a dummy OESELFTEST_METADATA section for
> > older data which doesn't have it. For example, the presence of meta_ide
> > tests indicates one particular type of test. Combined with the MACHINE
> > match, this should let us compare old and new data? That would mean
> > metadata_matches() would need to see into the actual results too.
>
> I thought about this possibility while starting to work on this (checking for
> existing metadata to kind of generate/guess test configuration), but I have
> felt
> that it would need too much "business logic" encoding in resulttool scripts,
> which could easily break if we want to change some tests configuration. I
> mean,
> if someone changes some parameters passed to oe-selftest in builders
> configuration, he will very probably not remember to update some "guess"
> filters
> in resulttool.
I think it will be important/useful to have good regression reports
against existing test results (e.g. ultimately on our LTS release
branches too). Whilst I don't normally like hardcoding things like
this, I think in this case the benefit outweighs the ugliness as long
as it "fades" over time which it will if we add proper metadata going
forward.
> I will start working on new revision of this series based on your comments
Sounds good, thanks!
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#177282):
https://lists.openembedded.org/g/openembedded-core/message/177282
Mute This Topic: https://lists.openembedded.org/mt/96964070/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
