Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

[Hideki Yamane]

Hi,

On Wed, 2 Dec 2020 11:28:06 +0100 Michael Prokop  wrote:
> Do we have any numbers for where xz resides in terms of
> decompression speed?

 As I've posted at salsa, its decompression is so slow.
 Its CPU is AMD Ryzen 5 PRO 3400GE (3.3GHz) but it takes 3 seconds.
 Older i686 CPUs would take more... but xz shrinks just 3 or 4MB.
 
https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/37#note_211939


-- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp

Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

[Michael Prokop]

* Hideki Yamane [Sun Nov 29, 2020 at 10:06:21AM +0900]:

> Now initramfs files are compressed with gzip
> But we can use other compression, i.e. LZO, LZ4 and ZSTD.
> Then which one is the best?

> * compression ratio
>ZSTD > GZIP > LZO > LZ4

> * decompression speed
>LZ4 > LZO = ZSTD > GZIP

> I suggest to choose ZSTD, instead of GZIP since
>  - better compression ratio than current GZIP
>  - also better decompresion speed than current GZIP

[...]

> Yes, 59MB initramfs file becomes 40MB (2/3)! plus bonus, better decompression 
> speed.

Thanks for bringing this up.

Do we have any numbers for where xz resides in terms of
decompression speed? Is it enough to just decompress the
files in userspace for getting numbers, or is the kernel doing
something special™ to be kept in mind?

JFTR, some further numbers regarding file sizes:

| # file netboot.initrd*
| netboot.initrd.gzip: gzip compressed data, last modified: Wed Dec  2 10:13:21 
2020, from Unix, original size 135559168
| netboot.initrd.lz4:  LZ4 compressed data (v0.1-v0.9)
| netboot.initrd.lzop: lzop compressed data - version 1.030, LZO1X-1, os: Unix
| netboot.initrd.xz:   XZ compressed data
| netboot.initrd.zstd: Zstandard compressed data (v0.8+), Dictionary ID: None
| # ls -lah netboot.initrd*
| -rw-r--r-- 1 root root 43M Dec  2 10:13 netboot.initrd.gzip
| -rw-r--r-- 1 root root 51M Dec  2 10:25 netboot.initrd.lz4
| -rw-r--r-- 1 root root 60M Dec  2 10:24 netboot.initrd.lzop
| -rw-r--r-- 1 root root 29M Dec  2 10:17 netboot.initrd.xz
| -rw-r--r-- 1 root root 32M Dec  2 10:16 netboot.initrd.zstd

regards
-mika-


signature.asc
Description: Digital signature

Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

[Hideki Yamane]

Package: initramfs-tools
Version: 0.139
Severity: normal
Tags: patch

Dear Maintainer,

Now initramfs files are compressed with gzip
But we can use other compression, i.e. LZO, LZ4 and ZSTD.
Then which one is the best?

* compression ratio
   ZSTD > GZIP > LZO > LZ4

* decompression speed
   LZ4 > LZO = ZSTD > GZIP

I suggest to choose ZSTD, instead of GZIP since
 - better compression ratio than current GZIP
 - also better decompresion speed than current GZIP


--
With gzip
--

$ file /boot/initrd.img-5.9.0-*
/boot/initrd.img-5.9.0-3-amd64: gzip compressed data, from Unix, original size 
modulo 2^32 197680640
/boot/initrd.img-5.9.0-4-amd64: gzip compressed data, from Unix, original size 
modulo 2^32 197594112

$ du -h /boot/initrd.img-5.9.0-*
59M /boot/initrd.img-5.9.0-3-amd64
59M /boot/initrd.img-5.9.0-4-amd64

--
With zstd
--

$ file /boot/initrd.img-5.9.0-*
/boot/initrd.img-5.9.0-3-amd64: Zstandard compressed data (v0.8+), Dictionary 
ID: None
/boot/initrd.img-5.9.0-4-amd64: Zstandard compressed data (v0.8+), Dictionary 
ID: None

$ du -h /boot/initrd.img-5.9.0-*
40M /boot/initrd.img-5.9.0-3-amd64
40M /boot/initrd.img-5.9.0-4-amd64
--

Yes, 59MB initramfs file becomes 40MB (2/3)! plus bonus, better decompression 
speed.


However, there's a problem to do so - zstd package's Priority is not standard 
one 
- it's "optional". 

> Package: gzip
> Version: 1.10-2
> Priority: required

> Package: zstd
> Version: 1.4.5+dfsg-4
> Priority: optional

It means there is not zstd package in every environment. And, just raise its 
Priority
means bloat minimal system size that is a problem we want to avoid.

Now initramfs-tools has a hack to avoid zstd (and other compression tools)
absence in mkinitramfs command.

>　if ! command -v "${compress}" >/dev/null 2>&1; then
> compress=gzip
> echo "No ${compress} in ${PATH}, using gzip"
> fi

Set COMPRESS=zstd in /etc/initramfs/initramfs.conf and no zstd installed,
there's no problem.

> $ sudo update-initramfs -u
> update-initramfs: Generating /boot/initrd.img-5.9.0-4-amd64
> No zstd in /usr/bin:/sbin:/bin, using gzip

 (abobe message is patched with #971270)


So, just set default compression as zstd can brings better result for many 
users,
IMO (Not sure raising zstd package Priority (to standard) is required or not).
>From 78c4fe32a447b0dc46045f0c8a32a4b246691711 Mon Sep 17 00:00:00 2001
From: Hideki Yamane 
Date: Sun, 29 Nov 2020 10:05:48 +0900
Subject: [PATCH] Use zstd as default compression for initramfs

---
 conf/initramfs.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/conf/initramfs.conf b/conf/initramfs.conf
index 01bdd85..a0c051b 100644
--- a/conf/initramfs.conf
+++ b/conf/initramfs.conf
@@ -41,7 +41,7 @@ KEYMAP=n
 # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
 #
 
-COMPRESS=gzip
+COMPRESS=zstd
 
 #
 # DEVICE: ...
-- 
2.29.2