On 1/27/23 10:51, Richard Purdie wrote: > On Fri, 2023-01-27 at 09:26 +0100, Alexis Lothoré wrote: >> On 1/26/23 23:42, Richard Purdie wrote: >>> On Tue, 2023-01-24 at 18:30 +0100, Alexis Lothoré via >>> lists.yoctoproject.org wrote: >>>> >>> Thanks. Since I like living dangerously, I've put these on the master >>> branch for the M2 release build. I wanted to see how this does in a >>> real world test. >> >> Thanks. Please feel free to reach back if you observe any unwanted side >> effect, >> since most of my knowledge is based on scripts reading and autobuilder builds >> history. I have tried to test it as much as possible on my machine, but as I >> was >> discussing with Alexandre Belloni, the many stubs and fake data used do not >> make >> the tests as accurate as the real world case. > > The result of the M2 build is here, specifically the regression report: > > https://autobuilder.yocto.io/pub/releases/yocto-4.2_M2.rc2/testresults/testresult-regressions-report.txt > > (warning, it is a 13MB file, which may hint at some problems) > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio > > was the log from running the script. It also suggests there may be a > few issues. > > I think we need some improvements to the report. Firstly it should > state at the top which revisions are being compared and any tags they > correspond to.
Yes, I was thinking about this while tweaking send-qa-email. Another point I need to understand is how the regression report is managed if resulttool finds multiple tags in results dir concerning the same yocto revision. I feel that the result is not the same if it finds one tag or multiple tags for the same revision. Does it really make sense to check regression against multiple tags referencing the same yocto revision ? If so, indeed we should make the tag name appear. If not, how to distinguish which tag is appropriate ? This question is more linked to resulttool internals than to send-qa-email. > > Next, we need to understand why the large chunk of tests "disappeared". > it may be related to the "Failed to retrieved" messages? > > I also happen to know the reproducibility tests failed in this build, > yet the regression report has no mention of that. ACK, I will take a look. > > So the patches are a start but there is much more to do. Regards, -- Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#59084): https://lists.yoctoproject.org/g/yocto/message/59084 Mute This Topic: https://lists.yoctoproject.org/mt/96502727/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
