On Mon, Sep 14, 2026 at 09:39:07AM -0700, Kees Cook wrote:
> On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote:
> > On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
> > 1.6s, and with pigz it takes 0.09s, so the performance increase is
> > significant.
>
> Neat! I wanna go learn how pigz accomplishes this -- I thought the
> problem with gzip was a common lookup table. Anyway...
Yeah not sure on the details :)
>
> > --- a/Documentation/kbuild/reproducible-builds.rst
> > +++ b/Documentation/kbuild/reproducible-builds.rst
> > @@ -76,6 +76,22 @@ include generated files. You should ensure the source
> > tree is
> > pristine by running ``make mrproper`` or ``git clean -d -f -x`` before
> > building a source package.
> >
> > +Compression tools
> > +-----------------
> > +
> > +The compressed kernel image, compressed modules and packages are produced
> > +using the program named by the make variable ``KGZIP`` (described in
> > +Documentation/kbuild/kbuild.rst).
> > +
> > +It defaults to ``pigz`` if installed (a parallel implementation of gzip),
> > +or ``gzip`` otherwise.
> > +
> > +The generated output between two invocations of identical builds with
> > +either of the default tools will be byte-for-byte equivalent.
> > +
> > +However, for reproducible builds, ensure the same tool is used on all build
> > +hosts, as different tools may generate different output from one another.
> > +
> > Module signing
> > --------------
> >
>
> This doc update seems totally unneeded? Having the same build tools for RB
> is already a known requirement. I don't think anything new is added here?
Ack that's fair enough, will drop the doc update.
>
> > diff --git a/Makefile b/Makefile
> > index 790ef23c5e8a..38c0cdc9f591 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -561,7 +561,7 @@ PERL = perl
> > PYTHON3 = python3
> > CHECK = sparse
> > BASH = bash
> > -KGZIP = gzip
> > +KGZIP := $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)
>
> I think the more idiomatic way to do this is:
>
> KGZIP := $(call try-run,command -v pigz,pigz,gzip)
try-run is defined in scripts/Makefile.compiler which is only included ~200
lines after KGZIP is set.
That'll also set up and tear down a temp dir for a probe that doesn't need
that, so I think it's fine as it is.
>
> However, parallelism needs to be set. We can't let it eat all CPUs: it
> needs to respect the -j make option (and make its CPU reservation known
> to "make"), which we already have a solution for in
> scripts/jobserver-exec.
The only place where it's invoked is vmlinux.bin at the end of the serial
tail, where all the tokens would be free anyway.
So I don't think it really buys anything at all?
Using jobserver-exec would also put python3 and two wrapper scripts in
front of every gzip in the build including tar -I "$(KGZIP)" when packaging
and some arm and m68k scripts too.
(For the module zips it's already constrained to a single process.)
Overall I think it's less complexity and really no delta to just invoke it
as normal.
It's designed as drop-in so it makes sense to use it as that.
>
> However, I would actually argue that given such an improvement we should just
> make pigz explicitly required and not optional. It is packaged everywhere:
>
> │ Debian / Ubuntu │ ✅ │ pigz (main)
> │ Fedora │ ✅ │ pigz 2.8 (current)
> │ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream
> │ openSUSE / SLE │ ✅ │ pigz
> │ Arch Linux │ ✅ │ pigz (extra)
> │ Alpine │ ✅ │ pigz
> │ Gentoo │ ✅ │ app-arch/pigz 2.8
>
> Only RHEL appears a little glitchy, but likely they would trivially move
> it to base since it's already packaged, but off in EPEL.
Definitely not something for this series, the fallback is one invocation of
command -v and I don't really want to break RHEL either :)
If, once this has landed, we want to go that way then it's simple enough
for us to change it.
>
> So, I would say that pigz would be best run as something like:
>
> KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec
> $(abs_srctree)/scripts/parallel-pigz
>
> with scripts/parallel-pigz being something like:
>
> #!/bin/sh
> exec pigz -p ${PARALLELISM:-1} "$@"
>
> > --- a/scripts/Makefile.modinst
> > +++ b/scripts/Makefile.modinst
> > @@ -145,8 +145,10 @@ endif
> > #
> > # Compression
> > #
> > +# Modules are compressed in parallel by make itself, so keep the compressor
> > +# single-threaded when it is pigz.
> > quiet_cmd_gzip = GZIP $@
> > - cmd_gzip = $(KGZIP) -n -f $<
> > + cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword
> > $(KGZIP)))),-p 1) -n -f $<
> > quiet_cmd_xz = XZ $@
>
> Then this could be:
>
> cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $<
This is already achieved for the modinst case with a one-liner in any case.
>
> and we patch scripts/jobserver-exec:
>
> diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
> index 21b319e6c9a5..8b953148ee9f 100755
> --- a/scripts/jobserver-exec
> +++ b/scripts/jobserver-exec
> @@ -5,6 +5,7 @@
> Determines how many parallel tasks "make" is expecting, as it is
> not exposed via any special variables, reserves them all, runs a subprocess
> with PARALLELISM environment variable set, and releases the jobs back again.
> +If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1.
>
> See:
>
> https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
> index 0b1ffdf9f7a3..fc38c5020b22 100755
> --- a/tools/lib/python/jobserver.py
> +++ b/tools/lib/python/jobserver.py
> @@ -29,6 +29,11 @@ $claim child to do the actual work.
> The end goal here is to keep the total number of build tasks under the
> limit established by the initial ``make -j$n_proc`` call.
>
> +Setting the ``SINGLE_THREADED`` environment variable skips the reservation
> +entirely and runs the command with ``PARALLELISM=1``. This is meant for
> callers
> +that run many short commands in parallel themselves, where each one should
> use
> +only the job slot it already holds.
> +
> See:
>
> https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> """
> @@ -68,6 +73,13 @@ class JobserverExec:
> self.is_open = True # We only try once
> self.claim = None
> #
> + # SINGLE_THREADED asks for no reservation at all: the command runs
> + # with PARALLELISM=1, using only the slot its caller already holds.
> + #
> + if os.environ.get('SINGLE_THREADED'):
> + self.claim = 1
> + return
> + #
> # Check the make flags for "--jobserver=R,W"
> # Note that GNU Make has used --jobserver-fds and --jobserver-auth
> # so this handles all of them.
>
> --
> Kees Cook
--
Cheers, Lorenzo
