Hi,
Op 10-01-2023 om 10:24 schreef Ferry Toth:
Hi,
On 10-01-2023 02:57, Randy MacLeod wrote:
Add Zheng.
Switch to Trevor's gmail since he might still be interested in this.
On 2023-01-09 05:49, Ferry Toth via lists.yoctoproject.org wrote:
Hi
On 09-01-2023 11:39, Alexander Kanavin wrote:
On Sun, 8 Jan 2023 at 23:13, Ferry Toth <[email protected]> wrote:
Now it works I'm not sure what to do. Richard marked the original
patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.
Or may be with a little work (from me) and a run on the CI servers we
could make it go in?
I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.
As there is already a clone of ninja available with jobserver support
available I added another patch to update to that version and make an
intercept to actually pass the named pipe to ninja.
Find it here: https://github.com/htot/poky/commits/kirkstone
This makes both make and ninja threads running from a shared thread
pool. As cmake relies on ninja recipies are covered as well.
I guess this takes care of the majority of recipes.
In my case this new patch shaves no more then 1 minutes from my image
build time (1h46) as the recipes built using ninja are not consuming
that much memory (like nodejs). I hope others will benefit more.
Todo: location of the pipe as mentioned by Richard
Zheng and I *started* on that for make over the Xmas holiday.
See the (poorly formatted) thread:
Add support for limiting CPU pressure, contrib, 2022/12/20
in:
https://lists.gnu.org/archive/html/bug-make/2022-12/threads.html
There were mixed reviews upstream with the maintainer, Paul Smith,
seeming to prefer that we investigate the job server approach and the
current
load averaging. I'll happily try to find time to play with Ferry's
job server work.
The work is actually Richard's. I only fixed what broke over time.
I've been thinking about the various workflows and as Richard said,
it seems
that for many people who only do one build at a time, the job server
and maybe
a little linker restraint, would be all that's needed. For activities
such
as YP AB, we could teach the main job server to also look at
/proc/pressure
as a way to limit the pool size we could make a meta-jobserver for
those who don't
want/need to worry about non-compile tasks such as tests and build
clean-up.
Note that cargo has the notion of a job server:
https://github.com/rust-lang/cargo/issues/1744
https://github.com/rust-lang/cargo/pull/4110
and ninja has an open issue:
https://github.com/ninja-build/ninja/issues/1139
and an initial implementation:
https://github.com/stefanb2/ninja/tree/topic-jobserver-fifo
What other build tools are in need of regulation and/or job server
patches?
What I read, gcc has already -flto=jobserver.
Other (single threaded CPU intensive) might just be started from
jobclient?
../Randy
Alex
Yeah, I'd rather teach the kernel to consider thrashing when
scheduling jobs. As is now any user process can slow down any other
users process and even the kernel itself to a standstill.
But kernel developers consider those issues "orthogonal" (i.e. they
don't want to make the scheduler aware of io).
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#58988): https://lists.yoctoproject.org/g/yocto/message/58988
Mute This Topic: https://lists.yoctoproject.org/mt/82015730/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
