On Mon, Sep 14, 2026 at 11:00:27AM -0700, Kees Cook wrote:
> On Mon, Sep 14, 2026 at 10:22:13AM +0100, Lorenzo Stoakes (ARM) wrote:
> > Module finalisation on allmodconfig builds consists of a large number of
> > very short-lived jobs, and the make job dispatcher cannot possibly dispatch
> > jobs fast enough.
>
> This "cannot possibly" sounds like a weird LLM language intensifier. I'd
Again that's my writing actually, probably best to stop assuming LLM now ;)
> rather a concrete description of the problem, not this kind of
> (redundant?) vagueness.
In the very next paragraph I say:
For allmodconfig x86-64 this can be on the order of ~22,000 jobs of
a few milliseconds in duration each.
You did also say that the commit messages were over-long, so there's a
trade-off here :)
But I think best to expand it a bit if it's not clear.
The general idea is that each job is so short (ms) that the work of
dispatching them exceeds the time doing the work, so you need to shard
things.
And the work of dispatching is heavy - each modfinal instance means it has
to process ~22k .cmd files of every .mod.o and .ko.
I will update the commit message to reflect this.
As a result, we need to batch these (see below).
>
> > [...]
> > Fix this by splitting modules.order into chunks of 128 at a time, run in
> > parallel.
>
> Why "128"? This seems tied to the 128-thread test machine, but ends up
Honestly Kees :) you really think I'd let a hardcoded-to-my-machine
variable through to the point of being called out in the commit msg? :P
No, that's not what this is.
> getting hard-coded, but this choice of value needs some rationale, IMO.
The rationale bit is fair enough, I thought it was somewhat implied but
it's a heuristically-determined value which determines how best to shard
the jobs.
So, it's about both getting parallelism and batching up to offset this job
dispatch overhead, there's naturally an equilibrium.
Emperically:
modules per chunk 32 64 128 256 512
wall time 4.05s 4.07s 4.05s 4.03s 4.18s
But in more detail, allmodconfig tree (~11k modules on x86-64), best of 2,
make modules with *.ko *.mod.o deleted:
modules per chunk instances wall
----------------- --------- ------
1 11171 10.22s
2 5586 6.72s
4 2793 4.97s
8 1397 4.26s
16 699 4.05s
32 350 4.05s
64 175 4.07s
128 88 4.05s
256 44 4.03s
512 22 4.18s
which I think makes things clearer.
So realistically 32 - 256 is the right sort of range. You also have to take
into account the fact that you might be building fewer modules.
The argument for 128 is that this is the mid-point of where the graph
flattens off for a larger number of modules.
For a smaller number, you're going to have a single dispatch or less and
the delta won't be that much anyway.
So it's very much a sensibly derived empirical value.
I'll update the commit message to give this rationale there.
>
> --
> Kees Cook
And for avoidance of doubt, it's ME replying to things :P I deal with a LOT
of AI slop in mm so am quite sensitive to doing things right here (TM).
--
Cheers, Lorenzo
