On Wed, Jul 19, 2023 at 11:49 AM Richard Purdie <richard.pur...@linuxfoundation.org> wrote: > > On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote: > > On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote: > > > Hello, > > > > > > I got some feedback from RP: > > > > > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt > > > > > > This shows warnings for duplicate tests and he also asks being where the > > > qemuarm64 went, both issues being probably related. > > > > I had a look and you probably need to add a: > > > > @OETestTag("toolchain-user") > > @OETestTag("runqemu") > > class RustSelfTestBase(RustSelfTestSystemEmulated): > > def test_check(self): > > self.test_rust() > > > > section to the tests to allow the tests to run for the non-IA > > architectures that use usermode emulation for the other tests which > > rust can't. > > > > That then just brings the question of why there are duplicate tests > > results being reported. This is where the result for an ID is being > > reported more than once. I haven't looked into why that might be > > happening. > > I really do want to get this rust test suite issue resolved so I went > digging into the code to find out what is really going on. > > Firstly, the duplicate test results. The issue is that you defined the > core class like this: > > class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): > def test_rust > > and python unittest has a convention where anything starting "test_" is > a test. > > This meant that the rust test ran unguarded in all the oe-selftest > targets on the autobuilder and not just in the toolchain-system > filtered section. > > The easiest fix is to drop the RustSelfTestBase class and move the > toolchain-system decorator to RustSelfTestSystemEmulated. That will > resolve the duplicate test warnings and ensure things run where they > should. You could have worked out this issue by finding that there were > rust test results in oe-selftest-* testresults.json files, e.g. here: > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json > > Moving on, the test result names really don't look good with the > "[ui] " and similar prefixes in the results file. I've patched a tweak > in to drop that. > > I also noticed that there were no skipped tests being reported in the > results. This was due to "SKIP" being used instead of "SKIPPED" which > resulttool looks for. > > We need to also add the toolchain-user decorator to make sure that the > tests run for the "user" architectures since we don't have any user > mode rust test equivalent. > > I've rolled all these changes into a patch on master-next: > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22 > > which I'll put into testing. > > Looking to the future, I have concerns about the ease of maintenance of > this huge patch to rust to disable failing tests. I'd propose we change > this to a hardcoded list of tests to ignore in the result parsing code > which will be easier to maintain in the future. >
That feels far more maintainable - the scale of the patch really concerned me. Also I guess if it's just a list, then running with everything enabled on upgrade to see what passes/fails would be a relatively easy option. > We may want to take that as a subsequent follow up patch. > > Cheers, > > Richard > > > -- Alex Kiernan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184579): https://lists.openembedded.org/g/openembedded-core/message/184579 Mute This Topic: https://lists.openembedded.org/mt/100232853/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
