On 10/28/22 11:37, Richard Purdie wrote: > On Fri, 2022-10-28 at 11:29 -0400, Sean Anderson wrote: >> On 10/28/22 11:09, Richard Purdie wrote: >> > On Wed, 2022-10-26 at 13:21 -0400, Sean Anderson wrote: >> > > As noted in the cover letter, I ran >> > > >> > > oe-selftest -r fitimage.FitImageTests >> > >> > Ok, good. That at least means you were only running one class of tests. >> > I was worried you were running all of them! >> > >> > > I also tried using -j$(nproc), but I saw no increase in parallelism >> > > outside of the usual for bitbake. This was especially noticable for >> > > do_rootfs, which is single-threaded. >> > >> > Sadly the parallelism works on a per test class basis so it wouldn't >> > help in this case. There are only small marginal gains from running >> > tests in individual build directories so we don't do that. >> >> I estimate it could have saved me 2-3 minutes every build, since it could >> have parallelized the root filesystem stuff. > > On an initial run, it could have also ended up building a lot of pieces > in parallel needlessly so it is all a bit of a compromise. It might be > worth looking into whether we can make that an option, off by default. > >> > > This is ommitted above, but I *had* to use -j1 in order to avoid >> > > manually wiping out my existing build directory each time (and instead >> > > ending up with dozens of pid-named directories). This is documented >> > > nowhere, and I found it in some old IRC logs. >> > >> > Parallelism using differently named build directories is an >> > implementation detail, not something which the -j option implies.I >> > guess you were also using --keep-builddir >> >> Failing builds don't remove the test directory so you can inspect the build >> output. As you might imagine, I had a lot of failing builds. > > I'm very familiar with that myself, yes. > > We did once used to reuse the build directory, that challenge is we > have no idea what the user has done in there prior to the test so it > potentially makes the test results potentially incorrect. > >> > > > We haven't really had anyone try and optimise the tests either, I'm >> > > > sure there will be things in there which can help. Please don't let the >> > > > speed put you off trying to improve things and extend our coverage! >> > > >> > > The poor speed of these self tests (and of everything related to the >> > > yocto project in general) makes this project frustrating to contribute >> > > to. It took me around 2 days to go from my prototype to this series, >> > > most of which was spent waiting for tests to compile and losing whatever >> > > train of thought I had. I probably went through perhaps 20 revisions. If >> > > I was working on e.g. U-Boot, I could have made 20 revisions in 2 hours, >> > > as it takes around 15 seconds to recompile it and run the full unit test >> > > suite. >> > > >> > > On the topic of these specific tests, part of the problem is that >> > > do_rootfs is a bottleneck which takes around 45-60s on my system. Every >> > > test which modifies something in the rootfs incurs this overhead. >> > >> > For better or worse we've 'a few' more moving pieces than U-Boot. >> > >> > Building a root filesystem from packages is a non-trivial task, taking >> > under a minute is in some ways pretty good already. The only other >> > thing we could do is incremental rootfs construction where it would >> > add/remove changed packages. I'd worry that the result may not always >> > be equal to a build from scratch and it might cause weird and >> > interesting reproducibility problems (particularly when you consider >> > things like postinsts). >> > >> > I would love to improve our development "iteration" time but I'm >> > struggling to see where we could get the speed gains from :(. Open to >> > other ideas... >> >> We don't have to build a full root filesystem. All of these tests just want >> e.g. an initramfs. An empty (or one file) filesystem would work just as well. >> If you still want to boot, you can make a busybox filesystem. > > Could we update the test just to use an initramfs then? > > I'm definitely a fan of keeping the tests as simple as we can whilst > still testing what we need to test.
I can look into this, but I'd prefer to do it as a follow-up to this series. I'll probably send a v2 later this week a fleshed-out commit message for patch 5/6 (and with it possibly all those variables moved to a separate bbclass to make it easier for other classes to create signed FITs). --Sean
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#172365): https://lists.openembedded.org/g/openembedded-core/message/172365 Mute This Topic: https://lists.openembedded.org/mt/94487631/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
