On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote:
> On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
> 1.6s, and with pigz it takes 0.09s, so the performance increase is
> significant.
Neat! I wanna go learn how pigz accomplishes this -- I thought the
problem with gzip was a common lookup table. Anyway...
> --- a/Documentation/kbuild/reproducible-builds.rst
> +++ b/Documentation/kbuild/reproducible-builds.rst
> @@ -76,6 +76,22 @@ include generated files. You should ensure the source
> tree is
> pristine by running ``make mrproper`` or ``git clean -d -f -x`` before
> building a source package.
>
> +Compression tools
> +-----------------
> +
> +The compressed kernel image, compressed modules and packages are produced
> +using the program named by the make variable ``KGZIP`` (described in
> +Documentation/kbuild/kbuild.rst).
> +
> +It defaults to ``pigz`` if installed (a parallel implementation of gzip),
> +or ``gzip`` otherwise.
> +
> +The generated output between two invocations of identical builds with
> +either of the default tools will be byte-for-byte equivalent.
> +
> +However, for reproducible builds, ensure the same tool is used on all build
> +hosts, as different tools may generate different output from one another.
> +
> Module signing
> --------------
>
This doc update seems totally unneeded? Having the same build tools for RB
is already a known requirement. I don't think anything new is added here?
> diff --git a/Makefile b/Makefile
> index 790ef23c5e8a..38c0cdc9f591 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -561,7 +561,7 @@ PERL = perl
> PYTHON3 = python3
> CHECK = sparse
> BASH = bash
> -KGZIP = gzip
> +KGZIP := $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)
I think the more idiomatic way to do this is:
KGZIP := $(call try-run,command -v pigz,pigz,gzip)
However, parallelism needs to be set. We can't let it eat all CPUs: it
needs to respect the -j make option (and make its CPU reservation known
to "make"), which we already have a solution for in
scripts/jobserver-exec.
However, I would actually argue that given such an improvement we should just
make pigz explicitly required and not optional. It is packaged everywhere:
│ Debian / Ubuntu │ ✅ │ pigz (main)
│ Fedora │ ✅ │ pigz 2.8 (current)
│ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream
│ openSUSE / SLE │ ✅ │ pigz
│ Arch Linux │ ✅ │ pigz (extra)
│ Alpine │ ✅ │ pigz
│ Gentoo │ ✅ │ app-arch/pigz 2.8
Only RHEL appears a little glitchy, but likely they would trivially move
it to base since it's already packaged, but off in EPEL.
So, I would say that pigz would be best run as something like:
KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec
$(abs_srctree)/scripts/parallel-pigz
with scripts/parallel-pigz being something like:
#!/bin/sh
exec pigz -p ${PARALLELISM:-1} "$@"
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -145,8 +145,10 @@ endif
> #
> # Compression
> #
> +# Modules are compressed in parallel by make itself, so keep the compressor
> +# single-threaded when it is pigz.
> quiet_cmd_gzip = GZIP $@
> - cmd_gzip = $(KGZIP) -n -f $<
> + cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword
> $(KGZIP)))),-p 1) -n -f $<
> quiet_cmd_xz = XZ $@
Then this could be:
cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $<
and we patch scripts/jobserver-exec:
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 21b319e6c9a5..8b953148ee9f 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -5,6 +5,7 @@
Determines how many parallel tasks "make" is expecting, as it is
not exposed via any special variables, reserves them all, runs a subprocess
with PARALLELISM environment variable set, and releases the jobs back again.
+If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1.
See:
https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
index 0b1ffdf9f7a3..fc38c5020b22 100755
--- a/tools/lib/python/jobserver.py
+++ b/tools/lib/python/jobserver.py
@@ -29,6 +29,11 @@ $claim child to do the actual work.
The end goal here is to keep the total number of build tasks under the
limit established by the initial ``make -j$n_proc`` call.
+Setting the ``SINGLE_THREADED`` environment variable skips the reservation
+entirely and runs the command with ``PARALLELISM=1``. This is meant for callers
+that run many short commands in parallel themselves, where each one should use
+only the job slot it already holds.
+
See:
https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
"""
@@ -68,6 +73,13 @@ class JobserverExec:
self.is_open = True # We only try once
self.claim = None
#
+ # SINGLE_THREADED asks for no reservation at all: the command runs
+ # with PARALLELISM=1, using only the slot its caller already holds.
+ #
+ if os.environ.get('SINGLE_THREADED'):
+ self.claim = 1
+ return
+ #
# Check the make flags for "--jobserver=R,W"
# Note that GNU Make has used --jobserver-fds and --jobserver-auth
# so this handles all of them.
--
Kees Cook
