Just to provide a high level overview of the ideas and proposals
coming from different sources for the requirements for testing and
validation of builds:
* Have terraform files for the testing infrastructure. Infrastructure
as code (IaC). Minus not emulated / nor cloud based, embedded
hardware. ("single command" replication of the testing infrastructure,
no manual steps).
* CI software based on Jenkins, unless someone thinks there's a better
alternative.
* Use autoscaling groups and improve staggered build + test steps to
achieve higher parallelism and shorter feedback times.
* Switch to a branching model based on stable master + integration
branch. PRs are merged into dev/integration which runs extended
nightly tests, which are
then merged into master, preferably in an automated way after
successful extended testing.
Master is always tested, and always buildable. Release branches or
tags in master as usual for releases.
* Build + test feedback time targeting less than 15 minutes.
(Currently a build in a 16x core takes 7m). This involves lot of
refactoring of tests, move expensive tests / big smoke tests to
nightlies on the integration branch, also tests on IoT devices / power
and performance regressions...
* Add code coverage and other quality metrics.
* Eliminate warnings and treat warnings as errors. We have spent time
tracking down "undefined behaviour" bugs that could have been caught
by compiler warnings.
Is there something I'm missing or additional things that come to your
mind that you would wish to add?
Pedro.
