On 11/8/18 1:42 AM, Richard Purdie wrote:
> There are going to be some changes for Yocto Project QA in the 2.7
> cycle. Intel is planning to scale back its involvement and wants to
> transition it to the wider community.
>
> Unfortunately we don't have people we can transition this to so a group
> of us have been analysing what the implications of this are and how we
> can best mitigate the changes.
>
> The short summary is don't panic. We will lose test coverage in some
> areas but we should be able to keep the core of the project strong and
> to the quality we're used to.
>
> For more info, read on.
>
> The first part of this exercise was to figure out what is being tested
> by the autobuilder and what is being tested by the QA teams today. The
> autobuilder piece is quite clear, it tests the configurations
> summarised by this config.json file:
>
> http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder-helper/tree/config.json
>
> Intel has slides showing what they test, I'm primarily interested in
> what the delta is between what gets tested by QA (manually or
> automatically) compared with the autobuilder. My summary of the delta
> in testing is:
>
> * Real hardware targets (Intel tests genericx86*, WindRiver tests
> beaglebone-yocto, mpc8315e-rdb and edgerouter)
>
> * manual OE-Core tests
>
> * oe-selftest on each supported distro
>
> * the other package manager backends (ipk + deb)
>
> * toaster
>
> * CROPS
>
> * eclipse-plugin
>
> * trigger runs of builder performance data for release builds
>
> We also have the complexity that the QA process has traditionally been
> tightly integrated into testopia (a plugin within bugzilla). We really
> need to drop testopia, its unmaintained holding us on an elderly
> bugzilla version. Testopia has two major functions, it contained all
> our testcases and handled the test execution, results handling and
> reporting.
>
> As a first step in moving forward I therefore asked the existing tests
> all be documented in lib/oeqa within OE-Core. Those patches landed late
> in 2.6, in particular with the population of the "manual" directory:
>
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/manual
>
> This now means OE-Core is definitive for the test cases we have (when
> looked at with the other selftest/runtime/sdk tests) and means we no
> longer need testopia for test case storage/management.
>
> For results tracking, I'm asking that builds that run tests generate a
> json results file containing the test results. This code merged late in
> 2.6:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=fc16be17878486181d460416618f4f06d2be40e3
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e73bcdd0590fbc82da515b37e5cf32726f231cf9
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d89e06083eeea1bebdaa64f809e7e0a328e282f5
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4ef72d556f706e301e5155f5f228c38431acafe6
>
> I also added code which connects up ptest results into the main results
> file:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=fcf55f58ae73b4e54ea853c816c0fef0f33ca46c
>
> What we don't have yet is results management, the results aren't
> collected from the autobuilder, or analysed. There are patches from Ee
> Peng with some proposed code to do this on the mailing list:
>
> http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html
>
> but this isn't ready for merging yet.
>
> Going back to the above list and taking each item in turn:
>
> * Real hardware targets - Intel and Wind River are going to continue
> testing these
>
> * Manual OE-Core tests - these are now documented in OE-Core. We need
> to go through and analyse and clean the ones which don't make sense or
> we don't need.and In order to run these going forward they will need
> to be automated and likely won't be run until this is done.
>
> * oe-selftest on each supported distro - the public autobuilder should
> be able to be configured to do this (currently it runs randomly on one
> distro).
>
> * the other package manager backends (ipk + deb) - there was a bug in
> the public autobuilder, this was fixed and is now being tested
> automatically.
>
> * toaster - David Reyna is looking into selenium automation but likely
> won't be tested without automation
13008
>
> * CROPS - no coverage plan
>
> * eclipse-plugin - no coverage plan
Do we want bugs to track any discussions regarding these two?
>
> This means there are a few things we need to work on. I've therefore
> created this rough action list of things we need to do:
>
>
> * Review the manual test cases and remove any which are obsolete or
> unneeded
13009
>
> * Create a list of manual test cases which are important and need to be
> automated (file an enhancement bug for each?)
13010 ( place holder bug)
>
> * Add ptest running targets to the autobuilder to start collecting
> ptest results for arm and x86 machines under qemu.
13011
>
> * Add functionality to the autobuilder to enable oe-selftest to run
> once per distro
13007
>
> * Handle automated results collection on the autobuilder
11583 will open a new bug if that one does not meet needs
>
> * Create/merge results aggregation and analysis tool
13012
>
> * Run results aggregation and analysis on the autobuilder
13013
>
> * Handle triggering and collection of build performance data for
> release builds
13014
>
> * Upgrade bugzilla to a new version and drop testopia (keep around an
> old instance for archive purposes?)
13015
>
> * How to run our test suites for boards like beaglebone remotely on a
> LAVA board farm?
13016
>
> * Can we create some mechanism to allow easier end developer testing on
> real hardware ('plug a board into their laptop and go')?
13017
Do you mean PnP QA ?
>
> * Separate logic on autobuilder for a lighter test run and a longer
> more comprehensive one
13018
>
> The above should all really become bugzilla items and I'd welcome help
> with working on them. I know some people are already looking at some of
> the items (e.g. Anibal on the LAVA pieces).
I think I got all of them. Let me know if I missed anything. They are
all in my name until we disposition them.
>
> There is also a complication in all this which is stable releases.
> Somehow we need to not just automate master but also automate our point
> release testing for the stable series. This was why it was important to
> get automated results collecting into 2.6. I'm thinking about this and
> some pieces like the autobuilder enhancements should work for older
> releases too. We may need to backport some "features" against our
> normal policy to ensure we continue to have good testing for them.
>
> The email is long enough. I'm sure there are pieces I've forgotten but
> I'll stop here and ask for comments, concerns and other feedback.
>
> Bottom line is, we shouldn't panic but we are going to need changes and
> unless people step up, some things will fall through cracks (eclipse
> plugin and crops in particular). I'm very open to help with any of the
> above.
If people Panic, then they need to participate.
-armin
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> [email protected]
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
_______________________________________________
Openembedded-architecture mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
