On 2022-12-21 11:24, Paul Smith wrote:
On Tue, 2022-12-20 at 16:59 -0500, Randy MacLeod wrote:
While we are doing development with Yocto Project (1), a source-
based Linux distro builder, we found that many instances of making
running in parallel with other work can overload even a large
many-core build machine. Existing mechanisms that are either not
system-wide (-j) or are too slow(-l has 1 minute averaging), so in
order to make optimal use of a multi-core system for some larger
tasks we need a new mechanism.
Can you clarify what version of GNU make you're using?
4.3 from ubuntu-22.04 but the build pool for Yocto
has ~15 distros of various vintages as well as being
able to use the version of make (4.4) that we carry on
the oe-core master branch:
https://layers.openembedded.org/layerindex/recipe/245/
http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-devtools/make/make_4.4.bb
For any work we do here, we'd use that master branch version rather than
the older host version.
There have been a number of efforts to improve the accuracy of -l and
it does not (any longer) use a simple load detection, for EXACTLY this
reason (the value is too coarse, particularly at startup).
Okay. I'll try to allocate time to see what happens with 4.4.
In the latest version GNU make 4.4, it will use the content of
/proc/loadavg (if it exists) to figure out how many jobs are currently
executing on the system and use that as a measurement of load.
If that's not available, we use getloadavg() but we do some math on it
to modify the load average based on the number of jobs we've started in
the last second. Of course, this also suffers from the problem that it
only knows about jobs that THIS INSTANCE of make has started in the
last second.
Anyway you can read more about this in the comments in the source code:
https://git.savannah.gnu.org/cgit/make.git/tree/src/job.c?h=4.4#n1953
Thanks, I missed the recent changes amongst the decades old code! ;-)
What tests were you running to check these improvements?
--
# Randy MacLeod
# Wind River Linux
