Re: Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules
On Fri, Apr 9, 2021 at 1:31 PM Sedat Dilek wrote: > > On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski wrote: > > > > I originally posted the patch in a different form [1] even before > > Masahiro's changes. > > I've been testing this solution since December last year and posted it in > > March this year, > > after I made sure everything was working fine. This patch was tested by > > Oleksandr and he also didn't report any objections. [2] > > > > Masahiro notified me about the planned changes [3] and asked me to resend > > this patch, adjusted to those changes, which I did. > > > > My current logs: > > > > lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz > > CONFIG_DEBUG_INFO=y > > # CONFIG_DEBUG_INFO_REDUCED is not set > > # CONFIG_DEBUG_INFO_COMPRESSED is not set > > # CONFIG_DEBUG_INFO_SPLIT is not set > > CONFIG_DEBUG_INFO_DWARF4=y > > CONFIG_DEBUG_INFO_BTF=y > > CONFIG_DEBUG_INFO_BTF_MODULES=y > > lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz > > CONFIG_MODULE_COMPRESS_ZSTD=y > > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 > > > > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the > > upstream, it's an additional patch I use. > > > > The only difference - I don't use clang. Maybe those who use will comment > > on this. > > I relied on the opinions of Oleksander and a dozen other users who reported > > no errors in using zstd module compression. > > > > [1] https://marc.info/?l=linux-kbuild=161710402402989=2 > > > > [2] https://marc.info/?l=linux-kbuild=161710503403517=2 > > > > [3] https://marc.info/?l=linux-kbuild=161780602730829=2 > > I am a big fan of ZSTD and have it as default in all available Linux > Kconfigs and Debian's initramfs-tools. > So, I am highly interested in getting this fixed. > > Unfortunately, I have thrown away my yesterday's Clang-LTO build and > switched to Clang-CFI with builddeb - should do handle the same way. > > I see three iwlwifi.ko (as an example): > > $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07 > drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB > relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with > debug_info, not stripped > > That iwlwifi.ko with debug-info is optimized when moving to > debian/linux-image-dbg directory: > > $ LC_ALL=C ll > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18 > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: > ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with > debug_info, not stripped > > And think it's shrunk down and included debian/linux-image directory: > > $ LC_ALL=C ll > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 694K Apr 9 11:18 > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: > ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped > > I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed. > Fact is the one with debug-info is done correctly. > Might be builddeb script needs a special treatment. > OK, I see (sorry Gmail truncates my paste of snippet). We need to add in this block a CONFIG_MODULE_COMPRESS_XXX handling: [ scripts/package/builddeb ] #159: if is_enabled CONFIG_MODULES; then ... #184: fi Maybe other scripts in scripts/package/ directory, too. What do you say Masahiro? I have to admit I never used any compression for kernel-modules before and after recent changes in . - Sedat - [1] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/tree/scripts/package/builddeb?h=kbuild#n159
Re: Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules
On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski wrote: > > I originally posted the patch in a different form [1] even before Masahiro's > changes. > I've been testing this solution since December last year and posted it in > March this year, > after I made sure everything was working fine. This patch was tested by > Oleksandr and he also didn't report any objections. [2] > > Masahiro notified me about the planned changes [3] and asked me to resend > this patch, adjusted to those changes, which I did. > > My current logs: > > lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz > CONFIG_DEBUG_INFO=y > # CONFIG_DEBUG_INFO_REDUCED is not set > # CONFIG_DEBUG_INFO_COMPRESSED is not set > # CONFIG_DEBUG_INFO_SPLIT is not set > CONFIG_DEBUG_INFO_DWARF4=y > CONFIG_DEBUG_INFO_BTF=y > CONFIG_DEBUG_INFO_BTF_MODULES=y > lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz > CONFIG_MODULE_COMPRESS_ZSTD=y > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 > > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the > upstream, it's an additional patch I use. > > The only difference - I don't use clang. Maybe those who use will comment on > this. > I relied on the opinions of Oleksander and a dozen other users who reported > no errors in using zstd module compression. > > [1] https://marc.info/?l=linux-kbuild=161710402402989=2 > > [2] https://marc.info/?l=linux-kbuild=161710503403517=2 > > [3] https://marc.info/?l=linux-kbuild=161780602730829=2 I am a big fan of ZSTD and have it as default in all available Linux Kconfigs and Debian's initramfs-tools. So, I am highly interested in getting this fixed. Unfortunately, I have thrown away my yesterday's Clang-LTO build and switched to Clang-CFI with builddeb - should do handle the same way. I see three iwlwifi.ko (as an example): $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07 drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with debug_info, not stripped That iwlwifi.ko with debug-info is optimized when moving to debian/linux-image-dbg directory: $ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18 debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with debug_info, not stripped And think it's shrunk down and included debian/linux-image directory: $ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 694K Apr 9 11:18 debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed. Fact is the one with debug-info is done correctly. Might be builddeb script needs a special treatment. - Sedat -
Re: Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules
On Thu, Apr 8, 2021 at 11:05 PM Piotr Gorski wrote: > > No, the --rm option is essential. xz and gzip have the --rm option built in > as opposed to zstd, which is why I used it. I've been using zstd module > compression since last december (although I set a different compression level > on mine) and everything works fine. Oleksandr also tested it at his place and > didn't report any objections. [ CC me I am not subscribed to linux-kbuild or linux-kernel ] [ CC Nick ] Unfortunately, I do not find my initial posting which has all information. I add the link to the thread on linux-kbuild ML. So, I gave you as much information as I have (linux-config, make-line etc.) and you write "everything works fine"? What do you mean by "everything" - different compressors and none? Is that working "fine"? What build environment do you use? Here: Debian/testing AMD64. Did you try with... CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_DWARF5=y CONFIG_MODULE_COMPRESS_ZSTD=y ...Kconfigs enabled? As said I use builddeb from scripts directory to generate my Debian packages. Any chance you can test with builddeb? I have enabled Clang-LTO Kconfig. Tried with Clang-LTO Kconfig? This worked *before* and *after*... kbuild: add support for zstd compressed modules kbuild: remove CONFIG_MODULE_COMPRESS (CC Nick as he is listed as a reviewer here) ... not within my build-environment. For me this is a *regression*. - Sedat - [1] https://marc.info/?t=16179091462=1=2
