On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote: > On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote: > > Hello, > > > > I got some feedback from RP: > > > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt > > > > This shows warnings for duplicate tests and he also asks being where the > > qemuarm64 went, both issues being probably related. > > I had a look and you probably need to add a: > > @OETestTag("toolchain-user") > @OETestTag("runqemu") > class RustSelfTestBase(RustSelfTestSystemEmulated): > def test_check(self): > self.test_rust() > > section to the tests to allow the tests to run for the non-IA > architectures that use usermode emulation for the other tests which > rust can't. > > That then just brings the question of why there are duplicate tests > results being reported. This is where the result for an ID is being > reported more than once. I haven't looked into why that might be > happening.
I really do want to get this rust test suite issue resolved so I went digging into the code to find out what is really going on. Firstly, the duplicate test results. The issue is that you defined the core class like this: class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): def test_rust and python unittest has a convention where anything starting "test_" is a test. This meant that the rust test ran unguarded in all the oe-selftest targets on the autobuilder and not just in the toolchain-system filtered section. The easiest fix is to drop the RustSelfTestBase class and move the toolchain-system decorator to RustSelfTestSystemEmulated. That will resolve the duplicate test warnings and ensure things run where they should. You could have worked out this issue by finding that there were rust test results in oe-selftest-* testresults.json files, e.g. here: http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json Moving on, the test result names really don't look good with theĀ "[ui] " and similar prefixes in the results file. I've patched a tweak in to drop that. I also noticed that there were no skipped tests being reported in the results. This was due to "SKIP" being used instead of "SKIPPED" which resulttool looks for. We need to also add the toolchain-user decorator to make sure that the tests run for the "user" architectures since we don't have any user mode rust test equivalent. I've rolled all these changes into a patch on master-next: https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22 which I'll put into testing. Looking to the future, I have concerns about the ease of maintenance of this huge patch to rust to disable failing tests. I'd propose we change this to a hardcoded list of tests to ignore in the result parsing code which will be easier to maintain in the future. We may want to take that as a subsequent follow up patch. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184576): https://lists.openembedded.org/g/openembedded-core/message/184576 Mute This Topic: https://lists.openembedded.org/mt/100232853/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-