Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
On Thu, 2023-10-19 at 14:28 +0200, Alexis Lothoré wrote: > Hi Ross, > On 10/19/23 13:04, Ross Burton wrote: > > > On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org > > > wrote: > > I just skimmed your report and have some feedback to hopefully make it > > easier to read in the future. > > > > I’d suggest sorting the output in order of importance. For example, this > > is a section that I really don’t care about: > > > > Match: sdk_core-image-sato_x86_64_fvp-base_20230910083055 > > sdk_core-image-sato_x86_64_fvp-base_20231017222150 > > > > Put those at the bottom, or even better collate them into a single section > > where there have been no changes. > > > > Similarly: > > > > Match: runtime_core-image-sato_qemux86_20230911011430 > > runtime_core-image-sato_qemux86_20231017223736 > > Additionally, 1 new test(s) is/are present > > > > I guess marginally more important than identical results, but “there are > > new tests that passed” isn’t very interesting. > > Indeed, not so useful and hiding the real content at the bottom > > > Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940 > > oeselftest_almalinux-9.2_qemux86-64_20231017221342 > > > > Should they have matched? The host distro doesn’t match and this matters > > for some of the tests, as some distros don’t support some of the selftests. > > In this case specifically, there are seven regressions and six of them are > > specific to the host changing, which has the side-effect of hiding the one > > actual regression. > > Yeah, that's a point I have been struggling with when starting to update those > tools. The initial assumption I have started working with, after discussing > the > matter with Richard (see [1]), is the following: > 1. MACHINE _must_ match between base and target > 2. Different HOSTS _can_ be cross-checked > > But the issue you are pointing tends to show it does not work well in some > cases. I will have to do some tests to see if dropping 2. reduce this noise > without loosing valuable data, or if we need to find something smarter What the autobuilder does is either runs one selftest on a random host for q-quick, or for a-full it will run five selftests, one for "centos", one for "arm", one for "ubuntu", one for "fedora" and one for "debian". Ideally we'd therefore compare debian to debian if we have any choice. It is a question of finding the closest matches. That is hard in code though whilst keeping it generic. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61415): https://lists.yoctoproject.org/g/yocto/message/61415 Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
Hi Ross,
On 10/19/23 13:04, Ross Burton wrote:
>> On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org
>> wrote:
> I just skimmed your report and have some feedback to hopefully make it easier
> to read in the future.
>
> I’d suggest sorting the output in order of importance. For example, this is
> a section that I really don’t care about:
>
> Match: sdk_core-image-sato_x86_64_fvp-base_20230910083055
> sdk_core-image-sato_x86_64_fvp-base_20231017222150
>
> Put those at the bottom, or even better collate them into a single section
> where there have been no changes.
>
> Similarly:
>
> Match: runtime_core-image-sato_qemux86_20230911011430
> runtime_core-image-sato_qemux86_20231017223736
> Additionally, 1 new test(s) is/are present
>
> I guess marginally more important than identical results, but “there are new
> tests that passed” isn’t very interesting.
Indeed, not so useful and hiding the real content at the bottom
> Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
> oeselftest_almalinux-9.2_qemux86-64_20231017221342
>
> Should they have matched? The host distro doesn’t match and this matters for
> some of the tests, as some distros don’t support some of the selftests. In
> this case specifically, there are seven regressions and six of them are
> specific to the host changing, which has the side-effect of hiding the one
> actual regression.
Yeah, that's a point I have been struggling with when starting to update those
tools. The initial assumption I have started working with, after discussing the
matter with Richard (see [1]), is the following:
1. MACHINE _must_ match between base and target
2. Different HOSTS _can_ be cross-checked
But the issue you are pointing tends to show it does not work well in some
cases. I will have to do some tests to see if dropping 2. reduce this noise
without loosing valuable data, or if we need to find something smarter
>
> The report then lists the first however many regressions before announcing
> the summary:
>
> (In total, 7134 regressions/status changes detected)
> Additionally, 7 previously failing test(s) is/are now passing
> Additionally, 4622 new test(s) is/are present
>
> The headline figure of 7134 regressions should be first, as that’s the most
> important data point in a skim of the report. List the summary first, and
> then the breakdown.
ACK
> Grouping the results would be interesting, because the list got truncated I
> can’t see easily if all 7134 regressions were in ptestresult.gcc-g++-user.*
> or if that was just the first 100 and the rest were other components.
> Breaking the ptest results up by the second level component would be
> interesting, if it said something like this then we’d be able to get a feel
> for what components have broken from the report.
>
> 7134 regressions detected.
>
> ptestresult.gcc-libstdc++-v3-user.30_threads/thread/native_handle/cancel.cc
> execution test: PASS -> FAIL
> [ say 10 results per component ]
> And 6123 more in ptestresult.gcc-libstdc++v3
> ptestresult.gcc-libgomp.libgomp.c++/ctor-10.C: UNSUPPORTED -> UNRESOLVED
Makes sense. I have posted this morning the series introducing the display limit
([2]), I can work on a v2 implementing the ptest specific display limit
> This one also caught my eye:
> Regression: runtime_core-image-sato_qemuppc_20230910082140
> runtime_core-image-sato_qemuppc_20231017222112
> systemd.SystemdJournalTests.test_systemd_boot_time: PASSED -> SKIPPED
> Additionally, 1 new test(s) is/are present
>
> Is that comparing a systemd test run with a sysvinit test run?
I think the comparison is relevant, both are bout systemd system (many systemd
tests are present and OK in runtime_core-image-sato_qemuppc_20231017222112
results). However I do not get why it is marked as skipped and not failed:
"systemd.SystemdJournalTests.test_systemd_boot_time": {
"duration": 2.3783957958221436,
"log": "Error when parsing time from boot string",
"status": "SKIPPED"
}
Maybe an issue in the corresponding runner ?
> Thanks for the work on the tool so far, this is a lot easier to read than the
> full reports!
>
> Ross
Thank you for having taken time to give some feedback !
[1] https://lists.yoctoproject.org/g/automated-testing/message/1216
[2]
https://lore.kernel.org/openembedded-core/20231019095352.25923-1-alexis.loth...@bootlin.com/
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61414): https://lists.yoctoproject.org/g/yocto/message/61414
Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
On Thu, 2023-10-19 at 10:36 +0200, Alexis Lothoré wrote: > Hello, > > On 10/18/23 08:16, Pokybuild User wrote: > > > > A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder > > and is available at: > > > > > > https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1 > > The regression report looks worryingly empty. It has been generated with > yocto-4.2 as comparison base. When taking a look at yocto-testresults content > for it, I only find a few test results (11, while it should be something > around > 300). I am not sure yet if we did some mistakes while re-uploading manually > missing tests results during the 4.2 cycle, or if tests results are indeed > suffering some issues for release builds (this is repeatable on different > 4.2.x > releases). I'd noticed that too, not sure what happened but it wasn't what I'd expected. It would be good to understand what went wrong. > So in the mean time, here is a manual regression report between 4.3_M3 and > yocto-4.3.rc1: > > https://pastebin.com/6QbfGstR > > The report has a "rate limit" applied for noisy regression (patch incoming) I was thinking about ideas like printing the regressions first, then the matches. I think Ross has some good feedback about how we can improve things. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61413): https://lists.yoctoproject.org/g/yocto/message/61413 Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
Hi Alexis, > On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org > wrote: > The regression report looks worryingly empty. It has been generated with > yocto-4.2 as comparison base. When taking a look at yocto-testresults content > for it, I only find a few test results (11, while it should be something > around > 300). I am not sure yet if we did some mistakes while re-uploading manually > missing tests results during the 4.2 cycle, or if tests results are indeed > suffering some issues for release builds (this is repeatable on different > 4.2.x > releases). > > So in the mean time, here is a manual regression report between 4.3_M3 and > yocto-4.3.rc1: > > https://pastebin.com/6QbfGstR > > The report has a "rate limit" applied for noisy regression (patch incoming) Thanks for that, much appreciated. I just skimmed your report and have some feedback to hopefully make it easier to read in the future. I’d suggest sorting the output in order of importance. For example, this is a section that I really don’t care about: Match: sdk_core-image-sato_x86_64_fvp-base_20230910083055 sdk_core-image-sato_x86_64_fvp-base_20231017222150 Put those at the bottom, or even better collate them into a single section where there have been no changes. Similarly: Match: runtime_core-image-sato_qemux86_20230911011430 runtime_core-image-sato_qemux86_20231017223736 Additionally, 1 new test(s) is/are present I guess marginally more important than identical results, but “there are new tests that passed” isn’t very interesting. Now on to the regressions. Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940 oeselftest_almalinux-9.2_qemux86-64_20231017221342 Should they have matched? The host distro doesn’t match and this matters for some of the tests, as some distros don’t support some of the selftests. In this case specifically, there are seven regressions and six of them are specific to the host changing, which has the side-effect of hiding the one actual regression. The report then lists the first however many regressions before announcing the summary: (In total, 7134 regressions/status changes detected) Additionally, 7 previously failing test(s) is/are now passing Additionally, 4622 new test(s) is/are present The headline figure of 7134 regressions should be first, as that’s the most important data point in a skim of the report. List the summary first, and then the breakdown. Grouping the results would be interesting, because the list got truncated I can’t see easily if all 7134 regressions were in ptestresult.gcc-g++-user.* or if that was just the first 100 and the rest were other components. Breaking the ptest results up by the second level component would be interesting, if it said something like this then we’d be able to get a feel for what components have broken from the report. 7134 regressions detected. ptestresult.gcc-libstdc++-v3-user.30_threads/thread/native_handle/cancel.cc execution test: PASS -> FAIL [ say 10 results per component ] And 6123 more in ptestresult.gcc-libstdc++v3 ptestresult.gcc-libgomp.libgomp.c++/ctor-10.C: UNSUPPORTED -> UNRESOLVED This one also caught my eye: Regression: runtime_core-image-sato_qemuppc_20230910082140 runtime_core-image-sato_qemuppc_20231017222112 systemd.SystemdJournalTests.test_systemd_boot_time: PASSED -> SKIPPED Additionally, 1 new test(s) is/are present Is that comparing a systemd test run with a sysvinit test run? Thanks for the work on the tool so far, this is a lot easier to read than the full reports! Ross -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61411): https://lists.yoctoproject.org/g/yocto/message/61411 Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
Hello, On 10/18/23 08:16, Pokybuild User wrote: > > A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder > and is available at: > > > https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1 The regression report looks worryingly empty. It has been generated with yocto-4.2 as comparison base. When taking a look at yocto-testresults content for it, I only find a few test results (11, while it should be something around 300). I am not sure yet if we did some mistakes while re-uploading manually missing tests results during the 4.2 cycle, or if tests results are indeed suffering some issues for release builds (this is repeatable on different 4.2.x releases). So in the mean time, here is a manual regression report between 4.3_M3 and yocto-4.3.rc1: https://pastebin.com/6QbfGstR The report has a "rate limit" applied for noisy regression (patch incoming) Kind regards, Alexis -- Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61408): https://lists.yoctoproject.org/g/yocto/message/61408 Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)
A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at: https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1 Build URL: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062 Build hash information: bitbake: 5419a8473d6d4cd1d01537de68ad8d72cf5be0b2 meta-agl: 4063b4f9a712e32337c1d9678b2f2481dde2a346 meta-arm: 3ed13d25a065f29bd46ee725c708d12ebc3f175a meta-aws: a30a2b66f1447dc5abdbca6c5de743e39c08b99b meta-intel: 1bca60610c597571769edc4a057a04bfdbd2f994 meta-mingw: 65ef95a74f6ae815f63f636ed53e140a26a014ce meta-openembedded: 35bcd8c6ddfb6bc8729d0006dab887afcc772ec9 meta-virtualization: 827092c2ec925ea3a024dcda9ccfd738e351e6ba oecore: 4f84537670020a8d902248479efa9f062089c0d3 poky: f65f100bc5379c3153ee00b2aa62ea5c9a66ec79 This is an automated message from the Yocto Project Autobuilder Git: git://git.yoctoproject.org/yocto-autobuilder2 Email: richard.pur...@linuxfoundation.org -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61385): https://lists.yoctoproject.org/g/yocto/message/61385 Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
