Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules
I originally posted the patch in a different form [1] even before Masahiro's changes. I've been testing this solution since December last year and posted it in March this year, after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2] Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did. My current logs: lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_DEBUG_INFO_COMPRESSED is not set # CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF4=y CONFIG_DEBUG_INFO_BTF=y CONFIG_DEBUG_INFO_BTF_MODULES=y lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz CONFIG_MODULE_COMPRESS_ZSTD=y CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use. The only difference - I don't use clang. Maybe those who use will comment on this. I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression. [1] https://marc.info/?l=linux-kbuild=161710402402989=2 [2] https://marc.info/?l=linux-kbuild=161710503403517=2 [3] https://marc.info/?l=linux-kbuild=161780602730829=2
Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules
No, the --rm option is essential. xz and gzip have the --rm option built in as opposed to zstd, which is why I used it. I've been using zstd module compression since last december (although I set a different compression level on mine) and everything works fine. Oleksandr also tested it at his place and didn't report any objections.
[PATCH v3] kbuild: add support for zstd compressed modules
kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. V2 -> V3 * Fix a typo V1 -> V2 * Rebuild against linux-kbuild tree Signed-off-by: Piotr Gorski --- init/Kconfig | 8 +++- scripts/Makefile.modinst | 6 ++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/init/Kconfig b/init/Kconfig index 510f6fcd9b7f..b5744d32c4df 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2242,7 +2242,7 @@ choice Please note that the tool used to load modules needs to support the corresponding algorithm. module-init-tools MAY support gzip, and kmod - MAY support gzip and xz. + MAY support gzip, xz and zstd. Your build system needs to provide the appropriate compression tool to compress the modules. @@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ Compress modules with XZ. The installed modules are suffixed with .ko.xz. +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + help + Compress modules with ZSTD. The installed modules are suffixed + with .ko.zst. + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 191408f7a91a..f9fa2a3808b2 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -21,6 +21,7 @@ endif suffix-y := suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz suffix-$(CONFIG_MODULE_COMPRESS_XZ):= .xz +suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules)) @@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP$@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< +quiet_cmd_zstd = ZSTD $@ + cmd_zstd = $(ZSTD) -T0 --rm -f -q $< $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(call cmd,gzip) @@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(dst)/%.ko.xz: $(dst)/%.ko FORCE $(call cmd,xz) +$(dst)/%.ko.zst: $(dst)/%.ko FORCE + $(call cmd,zstd) + PHONY += FORCE FORCE: -- 2.31.0.97.g1424303384
[PATCH] kbuild: add support for zstd compressed modules
kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. V1 -> V2 * Rebuild against linux-kbuild tree Signed-off-by: Piotr Gorski --- init/Kconfig | 8 +++- scripts/Makefile.modinst | 6 ++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/init/Kconfig b/init/Kconfig index 510f6fcd9b7f..b5744d32c4df 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2242,7 +2242,7 @@ choice Please note that the tool used to load modules needs to support the corresponding algorithm. module-init-tools MAY support gzip, and kmod - MAY support gzip and xz. + MAY support gzip, xz and zstd. Your build system needs to provide the appropriate compression tool to compress the modules. @@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ Compress modules with XZ. The installed modules are suffixed with .ko.xz. +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + help + Compress modules with ZSTD. The installed modules are suffixed + with .ko.zst. + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 191408f7a91a..90d8b2c4304e 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -21,6 +21,7 @@ endif suffix-y := suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz suffix-$(CONFIG_MODULE_COMPRESS_XZ):= .xz +suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules)) @@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP$@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< +quiet_cmd_xz = ZSTD $@ + cmd_xz = $(ZSTD) -T0 --rm -f -q $< $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(call cmd,gzip) @@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(dst)/%.ko.xz: $(dst)/%.ko FORCE $(call cmd,xz) +$(dst)/%.ko.zst: $(dst)/%.ko FORCE + $(call cmd,zstd) + PHONY += FORCE FORCE: -- 2.31.0.97.g1424303384
[PATCH] init: add support for zstd compressed modules
kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. Signed-off-by: Piotr Gorski --- Makefile | 7 +-- init/Kconfig | 9 ++--- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 5160ff8903c1..82f4f4cc2955 100644 --- a/Makefile +++ b/Makefile @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP export mod_strip_cmd # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP -# or CONFIG_MODULE_COMPRESS_XZ. +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. mod_compress_cmd = true ifdef CONFIG_MODULE_COMPRESS @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS ifdef CONFIG_MODULE_COMPRESS_XZ mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f endif # CONFIG_MODULE_COMPRESS_XZ + ifdef CONFIG_MODULE_COMPRESS_ZSTD +mod_compress_cmd = $(ZSTD) -T0 --rm -f -q + endif # CONFIG_MODULE_COMPRESS_ZSTD endif # CONFIG_MODULE_COMPRESS export mod_compress_cmd diff --git a/init/Kconfig b/init/Kconfig index 8c2cfd88f6ef..86a452bc2747 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS bool "Compress modules on installation" help - Compresses kernel modules when 'make modules_install' is run; gzip or - xz depending on "Compression algorithm" below. + Compresses kernel modules when 'make modules_install' is run; gzip, + xz, or zstd depending on "Compression algorithm" below. module-init-tools MAY support gzip, and kmod MAY support gzip and xz. @@ -2273,7 +2273,7 @@ choice This determines which sort of compression will be used during 'make modules_install'. - GZIP (default) and XZ are supported. + GZIP (default), XZ, and ZSTD are supported. config MODULE_COMPRESS_GZIP bool "GZIP" @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP config MODULE_COMPRESS_XZ bool "XZ" +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS -- 2.31.0.97.g1424303384
