On Sat, 2023-01-14 at 23:33 +0100, Ferry Toth wrote: > Hi, > > Op 10-01-2023 om 10:24 schreef Ferry Toth: > > Hi, > > > > On 10-01-2023 02:57, Randy MacLeod wrote: > > > Add Zheng. > > > Switch to Trevor's gmail since he might still be interested in > > > this. > > > > > > On 2023-01-09 05:49, Ferry Toth via lists.yoctoproject.org wrote: > > > > Hi > > > > > > > > On 09-01-2023 11:39, Alexander Kanavin wrote: > > > > > On Sun, 8 Jan 2023 at 23:13, Ferry Toth <[email protected]> > > > > > wrote: > > > > > > > > > > > Now it works I'm not sure what to do. Richard marked the > > > > > > original > > > > > > patch > > > > > > as WIP. Maybe it's not appropriate for including into poky? > > > > > > If so I > > > > > > wouldn't mind carrying it in meta-intel-edison. > > > > > > > > > > > > Or may be with a little work (from me) and a run on the CI > > > > > > servers we > > > > > > could make it go in? > > > > > I'd rather teach make/ninja upstream to watch memory > > > > > consumption > > > > > and/or host pressure directly (e.g. similar to how it handles > > > > > -l). And > > > > > offer any resulting patches to upstream first. > > > > As there is already a clone of ninja available with jobserver support > available I added another patch to update to that version and make an > intercept to actually pass the named pipe to ninja. > > Find it here: https://github.com/htot/poky/commits/kirkstone > > This makes both make and ninja threads running from a shared thread > pool. As cmake relies on ninja recipies are covered as well. > > I guess this takes care of the majority of recipes. > > In my case this new patch shaves no more then 1 minutes from my image > build time (1h46) as the recipes built using ninja are not consuming > that much memory (like nodejs). I hope others will benefit more. > > Todo: location of the pipe as mentioned by Richard
Hi Ferry et al., I've reworked the patches (once again...) into a global class that sets up the jobserver configuration: https://lore.kernel.org/openembedded-core/[email protected]/T/#t Since it's "just" a class, it can be carried in custom meta layers. Also, it supports an external jobserver, which can be handy when building multiple yocto confifgurations on i.e. a CI server... // Martin > > > Zheng and I *started* on that for make over the Xmas holiday. > > > See the (poorly formatted) thread: > > > Add support for limiting CPU pressure, contrib, 2022/12/20 > > > in: > > > https://lists.gnu.org/archive/html/bug-make/2022-12/threads.html > > > > > > There were mixed reviews upstream with the maintainer, Paul > > > Smith, > > > seeming to prefer that we investigate the job server approach and > > > the > > > current > > > load averaging. I'll happily try to find time to play with > > > Ferry's > > > job server work. > > > > > The work is actually Richard's. I only fixed what broke over time. > > > I've been thinking about the various workflows and as Richard > > > said, > > > it seems > > > that for many people who only do one build at a time, the job > > > server > > > and maybe > > > a little linker restraint, would be all that's needed. For > > > activities > > > such > > > as YP AB, we could teach the main job server to also look at > > > /proc/pressure > > > as a way to limit the pool size we could make a meta-jobserver > > > for > > > those who don't > > > want/need to worry about non-compile tasks such as tests and > > > build > > > clean-up. > > > > > > > > > Note that cargo has the notion of a job server: > > > https://github.com/rust-lang/cargo/issues/1744 > > > https://github.com/rust-lang/cargo/pull/4110 > > > > > > > > > and ninja has an open issue: > > > https://github.com/ninja-build/ninja/issues/1139 > > > and an initial implementation: > > > https://github.com/stefanb2/ninja/tree/topic-jobserver-fifo > > > > > > What other build tools are in need of regulation and/or job > > > server > > > patches? > > > > > What I read, gcc has already -flto=jobserver. > > > > Other (single threaded CPU intensive) might just be started from > > jobclient? > > > > > ../Randy > > > > > > > > > > > > > > > > Alex > > > > > > > > Yeah, I'd rather teach the kernel to consider thrashing when > > > > scheduling jobs. As is now any user process can slow down any > > > > other > > > > users process and even the kernel itself to a standstill. > > > > > > > > But kernel developers consider those issues "orthogonal" (i.e. > > > > they > > > > don't want to make the scheduler aware of io). > > > > > > > > > > > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#60903): https://lists.yoctoproject.org/g/yocto/message/60903 Mute This Topic: https://lists.yoctoproject.org/mt/82015730/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
