On 10/26/22 13:00, Richard Purdie wrote: > On Wed, 2022-10-26 at 12:49 -0400, Sean Anderson wrote: >> On 10/26/22 10:08, Richard Purdie wrote: >> > This looks like a pretty nice improvement, I've been worried about how >> > entangled all this was becoming. Is there anything better we could do >> > with testing of this code? We rely increasingly on the tests to spot >> > regressions and I doubt the testcases we have cover the code >> > complexity. >> >> FWIW, I found the test suite to be fairly helpful when debugging this >> whole process, if agonizingly slow. I found many bugs in my initial >> implementation which were uncovered by the test suites, >> >> The best way to extend the test suite would probably be to have QEMU try >> to boot using the Linux and U-Boot. This would help cases where all the >> artifacts are created correctly but e.g. the wrong U-Boot is packaged. >> However, I am rather loathe to add more test cases like this, because it >> already takes around 10 minutes (with sstate!) to run the tests. This >> basically kills the speed of iterative development. > > Out of interest how were you running the tests? Did you look at the > parallelism options? Were you just running specific tests or specific > suites of tests?
As noted in the cover letter, I ran oe-selftest -r fitimage.FitImageTests I also tried using -j$(nproc), but I saw no increase in parallelism outside of the usual for bitbake. This was especially noticable for do_rootfs, which is single-threaded. This is ommitted above, but I *had* to use -j1 in order to avoid manually wiping out my existing build directory each time (and instead ending up with dozens of pid-named directories). This is documented nowhere, and I found it in some old IRC logs. > I do think we need something like you describe and adding a linux+uboot > approach is something I've wanted to do for a long time. Usually the > time is taken on our automated CI rather than impacting users, unless > they're working in that area at which point the tests are hopefully > helpful. Which of course provides no incentive to reduce runtime. > We haven't really had anyone try and optimise the tests either, I'm > sure there will be things in there which can help. Please don't let the > speed put you off trying to improve things and extend our coverage! The poor speed of these self tests (and of everything related to the yocto project in general) makes this project frustrating to contribute to. It took me around 2 days to go from my prototype to this series, most of which was spent waiting for tests to compile and losing whatever train of thought I had. I probably went through perhaps 20 revisions. If I was working on e.g. U-Boot, I could have made 20 revisions in 2 hours, as it takes around 15 seconds to recompile it and run the full unit test suite. On the topic of these specific tests, part of the problem is that do_rootfs is a bottleneck which takes around 45-60s on my system. Every test which modifies something in the rootfs incurs this overhead. --Sean
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#172166): https://lists.openembedded.org/g/openembedded-core/message/172166 Mute This Topic: https://lists.openembedded.org/mt/94487631/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
