On September 14, 2026 9:39:07 AM PDT, Kees Cook <[email protected]> wrote: >On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote: >> On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes >> 1.6s, and with pigz it takes 0.09s, so the performance increase is >> significant. > >Neat! I wanna go learn how pigz accomplishes this -- I thought the >problem with gzip was a common lookup table. Anyway... > >> --- a/Documentation/kbuild/reproducible-builds.rst >> +++ b/Documentation/kbuild/reproducible-builds.rst >> @@ -76,6 +76,22 @@ include generated files. You should ensure the source >> tree is >> pristine by running ``make mrproper`` or ``git clean -d -f -x`` before >> building a source package. >> >> +Compression tools >> +----------------- >> + >> +The compressed kernel image, compressed modules and packages are produced >> +using the program named by the make variable ``KGZIP`` (described in >> +Documentation/kbuild/kbuild.rst). >> + >> +It defaults to ``pigz`` if installed (a parallel implementation of gzip), >> +or ``gzip`` otherwise. >> + >> +The generated output between two invocations of identical builds with >> +either of the default tools will be byte-for-byte equivalent. >> + >> +However, for reproducible builds, ensure the same tool is used on all build >> +hosts, as different tools may generate different output from one another. >> + >> Module signing >> -------------- >> > >This doc update seems totally unneeded? Having the same build tools for RB >is already a known requirement. I don't think anything new is added here? > >> diff --git a/Makefile b/Makefile >> index 790ef23c5e8a..38c0cdc9f591 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -561,7 +561,7 @@ PERL = perl >> PYTHON3 = python3 >> CHECK = sparse >> BASH = bash >> -KGZIP = gzip >> +KGZIP := $(if $(shell command -v pigz 2>/dev/null),pigz,gzip) > >I think the more idiomatic way to do this is: > >KGZIP := $(call try-run,command -v pigz,pigz,gzip) > >However, parallelism needs to be set. We can't let it eat all CPUs: it >needs to respect the -j make option (and make its CPU reservation known >to "make"), which we already have a solution for in >scripts/jobserver-exec. > >However, I would actually argue that given such an improvement we should just >make pigz explicitly required and not optional. It is packaged everywhere: > > │ Debian / Ubuntu │ ✅ │ pigz (main) > │ Fedora │ ✅ │ pigz 2.8 (current) > │ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream > │ openSUSE / SLE │ ✅ │ pigz > │ Arch Linux │ ✅ │ pigz (extra) > │ Alpine │ ✅ │ pigz > │ Gentoo │ ✅ │ app-arch/pigz 2.8 > >Only RHEL appears a little glitchy, but likely they would trivially move >it to base since it's already packaged, but off in EPEL. > >So, I would say that pigz would be best run as something like: > > KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec > $(abs_srctree)/scripts/parallel-pigz > >with scripts/parallel-pigz being something like: > > #!/bin/sh > exec pigz -p ${PARALLELISM:-1} "$@" > >> --- a/scripts/Makefile.modinst >> +++ b/scripts/Makefile.modinst >> @@ -145,8 +145,10 @@ endif >> # >> # Compression >> # >> +# Modules are compressed in parallel by make itself, so keep the compressor >> +# single-threaded when it is pigz. >> quiet_cmd_gzip = GZIP $@ >> - cmd_gzip = $(KGZIP) -n -f $< >> + cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword >> $(KGZIP)))),-p 1) -n -f $< >> quiet_cmd_xz = XZ $@ > >Then this could be: > > cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $< > >and we patch scripts/jobserver-exec: > >diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec >index 21b319e6c9a5..8b953148ee9f 100755 >--- a/scripts/jobserver-exec >+++ b/scripts/jobserver-exec >@@ -5,6 +5,7 @@ > Determines how many parallel tasks "make" is expecting, as it is > not exposed via any special variables, reserves them all, runs a subprocess > with PARALLELISM environment variable set, and releases the jobs back again. >+If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1. > > See: > > https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver >diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py >index 0b1ffdf9f7a3..fc38c5020b22 100755 >--- a/tools/lib/python/jobserver.py >+++ b/tools/lib/python/jobserver.py >@@ -29,6 +29,11 @@ $claim child to do the actual work. > The end goal here is to keep the total number of build tasks under the > limit established by the initial ``make -j$n_proc`` call. > >+Setting the ``SINGLE_THREADED`` environment variable skips the reservation >+entirely and runs the command with ``PARALLELISM=1``. This is meant for >callers >+that run many short commands in parallel themselves, where each one should use >+only the job slot it already holds. >+ > See: > > https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver > """ >@@ -68,6 +73,13 @@ class JobserverExec: > self.is_open = True # We only try once > self.claim = None > # >+ # SINGLE_THREADED asks for no reservation at all: the command runs >+ # with PARALLELISM=1, using only the slot its caller already holds. >+ # >+ if os.environ.get('SINGLE_THREADED'): >+ self.claim = 1 >+ return >+ # > # Check the make flags for "--jobserver=R,W" > # Note that GNU Make has used --jobserver-fds and --jobserver-auth > # so this handles all of them. >
If we are going to add dependencies, can we add *any* scripting language that is less stone tablet than POSIX awk?
