I wanted to mention we now have a few developments working in our QA process/tooling workflow.
----------------------------------------------------------------------- In summary: We have buildhistory data comparing -next/mut builds to master: https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemux86/buildhistory.txt We have a summary of the test results (including ptest): https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-report.txt We have build performance reports: https://autobuilder.yocto.io/pub/non-release/20190224-3/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_master_20190224030033_47eb3d0.html All this data is stored into git repos: http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/ http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/ http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ ----------------------------------------------------------------------- In detail: Non-release builds publish some data to https://autobuilder.yocto.io/pub/non-release/YYYYMMDD-NN e.g. https://autobuilder.yocto.io/pub/non-release/20190223-15 where NN is an incremented number for each build. For the qemu machines, buildhistory is generated, stored in a git repo and buildhistory-diff is run against it. For -next/mut builds this is against master: https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemux86/buildhistory.txt https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemuarm/buildhistory.txt (for master it would be against the last master build) Historical data is saved into a git repo: http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/ (master-next and mut branches there will be force pushed) The structure also contains various testresults.json files. resulttool is used to find them all and import them into http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ We then run resulttool report to generate: https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-report.txt and we also generate the regression report: https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-regressions-report.txt although right now this isn't working as well as I'd like (since the previous build was a-quick, not an a-full and therefore there are missing components. The test results report includes ptest data and includes data on how long the ptests take to run and whether they timed out (a T in the time column). The build performance measurements are being run every 6 hours on two performance measurement workers (ubuntu1604 and centos7) triggered from the autobuilder. These generate the reports mentioned above as well as pushing the results data into: http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/ build performance tests can be run against custom branches to test code changes and then the report will be against the nearest commit in master. In a release build, the same data is generated but its published into: https://autobuilder.yocto.io/pub/releases/ The hope is that these reports and data storage allow us to much more accurately and easily figure out what was tested and whether/where we have regressions. It sounds relatively simple when put like this but there is a lot of work that has gone into this by many people both past (Markus' work on build performance or Paul on buildhistory) and present (Ee Peng on resulttool). It is great to finally see it being pulled together into something which is part of every build! There are no doubt bugs and things to improve but believe this to be in a good state for our 2.7 feature freeze. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core