On Thu, 14 Dec 2023 at 10:50, Richard Purdie <[email protected]> wrote: > Once a given build has run through the system, things do run much > faster but this is basically the performance issue I've been mentioning > in the weekly status reports. Even the above packageconfig change in > this patch would actually speed a lot of these up, but you're objecting > to that.
I no longer object :) the above explanations are fine with me. I did run the benchmarks though, so for the sake of having a fuller picture: default poky, core-image-minimal: 5134 tasks real 26m26.874s poky with this patch, core-image-minimal 4051 tasks real 25m53.513s poky without ptests, core-image-minimal 2997 tasks real 21m23.684s > The tests themselves are actually quite valuable as we're way beyond > the point I can work out which patch will break which features. Some of > the tests could undoubtedly be improved. If we disable the ptests for > the selftests, we run the risk of not reusing sstate so a better > question might be, why are all the tests not reusing sstate more > efficiently? Looking at that a-full again, it was abelloni's master-next which had something that triggered a mass rebuild, e.g. qemux86-64: Sstate summary: Wanted 8139 Local 2083 Mirrors 0 Missed 6056 Current 0 (25% match, 0% complete) (and that took over 5 hours to fulfil: https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/8224 ) My theory is that when all of the builders start at the same time in that situation, they will all rebuild the same things, and become very slow and overloaded. Including image builds in selftest as well, as it's parallelized in itself. I have a proposal: splitting a-full into stages. First, run the build steps from qemux86-64 and qemuarm64 (and arm64 on armhost), which would serve two purposes: - pre-populate sstate for all the the builders that will run in the next stage, accelerating them. It's possible some of the builders would still do full rebuilds, but then it becomes possible to look separately at why they can't reuse the sstate from the first stage. - do a quick, lightweight smoke check on the changes under test: if something breaks the build really badly, it will be caught there and then, the build will stop, and unneeded AB overload will be avoided. Which will in turn speed up everything else that's running at the same time. Alex
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192359): https://lists.openembedded.org/g/openembedded-core/message/192359 Mute This Topic: https://lists.openembedded.org/mt/103146402/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
