Re: [PATCH v10 4/8] usr: add support for zstd compressed initramfs
On Tue, Aug 04, 2020 at 09:25:23AM +0200, Sedat Dilek wrote: > On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven > wrote: > > On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell wrote: > > > From: Nick Terrell > > > * Add support for a zstd compressed initramfs. > > > * Add compression for compressing built-in initramfs with zstd. > > > --- a/usr/Kconfig > > > +++ b/usr/Kconfig > > > @@ -100,6 +100,15 @@ config RD_LZ4 > > > Support loading of a LZ4 encoded initial ramdisk or cpio buffer > > > If unsure, say N. > > > > > > +config RD_ZSTD > > > + bool "Support initial ramdisk/ramfs compressed using ZSTD" > > > + default y > > > + depends on BLK_DEV_INITRD > > > + select DECOMPRESS_ZSTD > > > + help > > > + Support loading of a ZSTD encoded initial ramdisk or cpio > > > buffer. > > > + If unsure, say N. > > > > I'm aware you copied this from the other entries, but IMHO "default y", > > and "If unsure, say N" are not a good combination. > you are right - for new stuff it should be "default n". It got already applied to Linus' tree with "y", and I think it'd be nice to have it as a default. Let's disable other compressors instead. On the other hand, having an unsupported rd compressor results in a boot failure that's not immediately obvious, so that's a reason for keeping the setting as "y". On the third hand, distributions default to either gz or xz, thus I'd say: * let's have gz xz zstd default to y, all others to n * drop bzip2 lzma1 completely * distros can't switch the mkinitramfs default yet, but if RD_ZSTD=y now, they'll be able to once they drop support for old kernels in a few years > What I am missing - still - is a note - that your user-space should > have the correct bits to support zstd-initramfs. > Unsure where to place such an information. Looks like INITRAMFS_COMPRESSION_* have lengthy prose but are not shown in menuconfig, while RD_*, with no such prose, are shown. The prose itself is grossly obsolete, too. I have some updates in: https://github.com/kilobyte/linux/commits/nobz2-v3 but that patchset needs rebasing and refreshing. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋⠀ It's time to migrate your Imaginary Protocol from version 4i to 6i. ⠈⠳⣄
Re: [PATCH v10 4/8] usr: add support for zstd compressed initramfs
On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven wrote:
>
> Hi Nick,
>
> On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell wrote:
> > From: Nick Terrell
> >
> > * Add support for a zstd compressed initramfs.
> > * Add compression for compressing built-in initramfs with zstd.
> >
> > I have tested this patch by boot testing with buildroot and QEMU.
> > Specifically, I booted the kernel with both a zstd and gzip compressed
> > initramfs, both built into the kernel and separate. I ensured that the
> > correct compression algorithm was used. I tested on arm, aarch64, i386,
> > and x86_64.
> >
> > This patch has been tested in production on aarch64 and x86_64 devices.
> >
> > Additionally, I have performance measurements from internal use in
> > production. On an aarch64 device we saw 19 second boot time improvement
> > from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64
> > device we saw a 9 second boot time reduction from switching from xz to
> > zstd.
> >
> > Reviewed-by: Kees Cook
> > Tested-by: Sedat Dilek
> > Signed-off-by: Nick Terrell
>
> Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add
> support for zstd compressed initramfs").
>
> > --- a/usr/Kconfig
> > +++ b/usr/Kconfig
> > @@ -100,6 +100,15 @@ config RD_LZ4
> > Support loading of a LZ4 encoded initial ramdisk or cpio buffer
> > If unsure, say N.
> >
> > +config RD_ZSTD
> > + bool "Support initial ramdisk/ramfs compressed using ZSTD"
> > + default y
> > + depends on BLK_DEV_INITRD
> > + select DECOMPRESS_ZSTD
> > + help
> > + Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
> > + If unsure, say N.
>
> I'm aware you copied this from the other entries, but IMHO "default y",
> and "If unsure, say N" are not a good combination.
>
Hi Geert,
you are right - for new stuff it should be "default n".
What I am missing - still - is a note - that your user-space should
have the correct bits to support zstd-initramfs.
Unsure where to place such an information.
If you send a patch for above feel free to add my:
Reviewed-by: Sedat Dilek
Thanks.
Regards,
- Sedat -
[1] https://bugs.debian.org/955469
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 --
> ge...@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like
> that.
> -- Linus Torvalds
Re: [PATCH v10 4/8] usr: add support for zstd compressed initramfs
Hi Nick,
On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell wrote:
> From: Nick Terrell
>
> * Add support for a zstd compressed initramfs.
> * Add compression for compressing built-in initramfs with zstd.
>
> I have tested this patch by boot testing with buildroot and QEMU.
> Specifically, I booted the kernel with both a zstd and gzip compressed
> initramfs, both built into the kernel and separate. I ensured that the
> correct compression algorithm was used. I tested on arm, aarch64, i386,
> and x86_64.
>
> This patch has been tested in production on aarch64 and x86_64 devices.
>
> Additionally, I have performance measurements from internal use in
> production. On an aarch64 device we saw 19 second boot time improvement
> from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64
> device we saw a 9 second boot time reduction from switching from xz to
> zstd.
>
> Reviewed-by: Kees Cook
> Tested-by: Sedat Dilek
> Signed-off-by: Nick Terrell
Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add
support for zstd compressed initramfs").
> --- a/usr/Kconfig
> +++ b/usr/Kconfig
> @@ -100,6 +100,15 @@ config RD_LZ4
> Support loading of a LZ4 encoded initial ramdisk or cpio buffer
> If unsure, say N.
>
> +config RD_ZSTD
> + bool "Support initial ramdisk/ramfs compressed using ZSTD"
> + default y
> + depends on BLK_DEV_INITRD
> + select DECOMPRESS_ZSTD
> + help
> + Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
> + If unsure, say N.
I'm aware you copied this from the other entries, but IMHO "default y",
and "If unsure, say N" are not a good combination.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
[PATCH v10 4/8] usr: add support for zstd compressed initramfs
From: Nick Terrell * Add support for a zstd compressed initramfs. * Add compression for compressing built-in initramfs with zstd. I have tested this patch by boot testing with buildroot and QEMU. Specifically, I booted the kernel with both a zstd and gzip compressed initramfs, both built into the kernel and separate. I ensured that the correct compression algorithm was used. I tested on arm, aarch64, i386, and x86_64. This patch has been tested in production on aarch64 and x86_64 devices. Additionally, I have performance measurements from internal use in production. On an aarch64 device we saw 19 second boot time improvement from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64 device we saw a 9 second boot time reduction from switching from xz to zstd. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- usr/Kconfig | 20 usr/Makefile | 1 + 2 files changed, 21 insertions(+) diff --git a/usr/Kconfig b/usr/Kconfig index 96afb03b65f9..2599bc21c1b2 100644 --- a/usr/Kconfig +++ b/usr/Kconfig @@ -100,6 +100,15 @@ config RD_LZ4 Support loading of a LZ4 encoded initial ramdisk or cpio buffer If unsure, say N. +config RD_ZSTD + bool "Support initial ramdisk/ramfs compressed using ZSTD" + default y + depends on BLK_DEV_INITRD + select DECOMPRESS_ZSTD + help + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. + If unsure, say N. + choice prompt "Built-in initramfs compression mode" depends on INITRAMFS_SOURCE != "" @@ -196,6 +205,17 @@ config INITRAMFS_COMPRESSION_LZ4 If you choose this, keep in mind that most distros don't provide lz4 by default which could cause a build failure. +config INITRAMFS_COMPRESSION_ZSTD + bool "ZSTD" + depends on RD_ZSTD + help + ZSTD is a compression algorithm targeting intermediate compression + with fast decompression speed. It will compress better than GZIP and + decompress around the same speed as LZO, but slower than LZ4. + + If you choose this, keep in mind that you may need to install the zstd + tool to be able to compress the initram. + config INITRAMFS_COMPRESSION_NONE bool "None" help diff --git a/usr/Makefile b/usr/Makefile index c12e6b15ce72..b1a81a40eab1 100644 --- a/usr/Makefile +++ b/usr/Makefile @@ -15,6 +15,7 @@ compress-$(CONFIG_INITRAMFS_COMPRESSION_LZMA) := lzma compress-$(CONFIG_INITRAMFS_COMPRESSION_XZ):= xzmisc compress-$(CONFIG_INITRAMFS_COMPRESSION_LZO) := lzo compress-$(CONFIG_INITRAMFS_COMPRESSION_LZ4) := lz4 +compress-$(CONFIG_INITRAMFS_COMPRESSION_ZSTD) := zstd obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o -- 2.27.0
