Re: Reducing initramfs size and speed up the generation
On Wed, Aug 09, 2023 at 05:30:04PM +, Benjamin Drung wrote: > > The benefits to this are that the firmware and base initrds only need be > > generated once regardless of number of kernels installed. And their > > generation is decoupled from kernel upgrades/installs and each other. > > So the firmware initrd only needs to be regenerated when the firmware > > package is upgraded, and that need not trigger the base initrd to be > > regenerated. Likewise, upgrading cryptsetup (or any other dependency of > > the base initrd) need not cause the firmware initrd to be regenerated. > > This approach could also be used with the early init microcode parts of > > the initrd. > This is an interesting idea. It raises some questions. > The list of firmware files to include in the initramfs is derived from > the kernel modules. Different kernel versions can require different > firmware files. How should that be handled with this approach? While it might be nice to further reduce the space requirements for /boot (especially in the face of ever-growing kernels+firmware), this question is precisely why I don't consider this viable. One of the properties of the current system is that installing new versions of packages that trigger regeneration of the initramfs for the most recent kernel should by default leave the initramfs for other kernels unmodified; this way, in the event of a regression, the last-known-good kernel can still be booted for recovery. If all of the kernels installed would be pointed at a common firmware initramfs that's pulled in by GRUB, then even in the case that the updated firmware package is *supposed* to be compatible with all the kernels on the system, it nevertheless loses this property that we haven't tampered with the last-known-good kernel and makes the system less resilient. We should prioritize resilience of boot recovery over reducing the size of /boot contents. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
Hi, On Wed, 2023-08-09 at 01:22 -0500, Glenn Washburn wrote: > Hi, > > I recently came across the thread last month of the same name as the > subject and would like to bring up for discussion some ideas orthagonal > to all the ones presented there. > > Some observations: > 1. I have observed that /lib/firmware in a sample initrd extracted on > my system is about 25% of the uncompressed archive. > 2. In that same archive, 13% of the archive was occupied by files not > in /lib/firmware nor /lib/modules > 3. I have currently many kernels installed on my system (really more > than I reasonably need), which means many initrds with duplicate > firmware files. > 4. GRUB supports concatenating initrds at boot time. > 5. GRUB can generate initrds from file paths (only regular files with > 0777 permissions). > > So the first idea is that the data from (1) and (2) is replicated > across initrds and can ideally be shared amongst all initrds. This > could be implemented by having initramfs-tools generate a common > /boot/initrd-firmware and /boot/initrd-common files which will contain > the firmware files and base initrd system files to be shared amongst all > kernel specific initrds. Then GRUB will at boot time combine the > initrds together with a command like: > > initrd /boot/initrd.img-kver-generic /boot/initrd-firmware \ > /boot/initrd-common > > This behavior, while until recently undocumented, has been available > for at least a decade, which might be a concern for older GRUB builds > using a config generated in this way. > > The benefits to this are that the firmware and base initrds only need be > generated once regardless of number of kernels installed. And their > generation is decoupled from kernel upgrades/installs and each other. > So the firmware initrd only needs to be regenerated when the firmware > package is upgraded, and that need not trigger the base initrd to be > regenerated. Likewise, upgrading cryptsetup (or any other dependency of > the base initrd) need not cause the firmware initrd to be regenerated. > This approach could also be used with the early init microcode parts of > the initrd. This is an interesting idea. It raises some questions. The list of firmware files to include in the initramfs is derived from the kernel modules. Different kernel versions can require different firmware files. How should that be handled with this approach? As you mentioned, GRUB might not be the only consumer of the initramfs. How should the non-GRUB use case be supported? -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reducing initramfs size and speed up the generation
Hi, I recently came across the thread last month of the same name as the subject and would like to bring up for discussion some ideas orthagonal to all the ones presented there. Some observations: 1. I have observed that /lib/firmware in a sample initrd extracted on my system is about 25% of the uncompressed archive. 2. In that same archive, 13% of the archive was occupied by files not in /lib/firmware nor /lib/modules 3. I have currently many kernels installed on my system (really more than I reasonably need), which means many initrds with duplicate firmware files. 4. GRUB supports concatenating initrds at boot time. 5. GRUB can generate initrds from file paths (only regular files with 0777 permissions). So the first idea is that the data from (1) and (2) is replicated across initrds and can ideally be shared amongst all initrds. This could be implemented by having initramfs-tools generate a common /boot/initrd-firmware and /boot/initrd-common files which will contain the firmware files and base initrd system files to be shared amongst all kernel specific initrds. Then GRUB will at boot time combine the initrds together with a command like: initrd /boot/initrd.img-kver-generic /boot/initrd-firmware \ /boot/initrd-common This behavior, while until recently undocumented, has been available for at least a decade, which might be a concern for older GRUB builds using a config generated in this way. The benefits to this are that the firmware and base initrds only need be generated once regardless of number of kernels installed. And their generation is decoupled from kernel upgrades/installs and each other. So the firmware initrd only needs to be regenerated when the firmware package is upgraded, and that need not trigger the base initrd to be regenerated. Likewise, upgrading cryptsetup (or any other dependency of the base initrd) need not cause the firmware initrd to be regenerated. This approach could also be used with the early init microcode parts of the initrd. And even more radical approach to the firmware is to use point (5) and have GRUB generate /lib/firmware at boot time. This would mean that the firmware is always up to date with what is on the system to be booted and both the disk space and generation time could be saved. This could also be applied to the kernel modules to have them be generated on the fly as well, saving the bulk of space and time used by initrd generation. GRUB could also do on the fly decompression of on disk compressed firmware modules that older kernels might not support (although sounds like this might not be much of an issue if zstd is back ported to the older kernel). The caveat to this second approach is that the way GRUB currently builds an initrd at run time is by passing a special argument to initrd for *each* file path in the initrd. So for my sample initrd above, that would be at least 921 arguments to initrd, one for each firmware file, plus the other initrds. I've not looked to see if this could be a problem, but it might. A better approach would be to allow GRUB to take a directory and recursively graft that tree into a point in the initrd tree (much like -graft-points of genisoimage). Also, this assumes that files that initrds are generated from must be accessible to GRUB. This might not be true in certain situations due to lack of GRUB support, eg. the firmware files are EXT4 encrypted files, but I suspect its the overwhelmingly common case. Another issue I see with these two approaches is they rely on GRUB, which might not be the bootloader of choice for Ubuntu on some architectures and the bootloaders there may not support these features. I think its reasonable to consider these approaches just for platforms supporting GRUB. Glenn -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 22 Jul 2023 at 02:03, Steve Langasek wrote: > > On Tue, Jul 11, 2023 at 01:17:27AM +0200, Benjamin Drung wrote: > > If you want to test it yourself, you can find initramfs-tools > > 0.142ubuntu7bd2 for mantic in my PPA: > > https://launchpad.net/~bdrung/+archive/ubuntu/ppa > > What blocks this from being uploaded to mantic? AFAICS the behavior should > remain unchanged for existing kernel+firmware packages, and it's therefore > safe to push more widely. > I have reviewed this, and I approve of these changes. Please note, that we need this uploaded in mantic, such that linux-firmware can declare breaks on versions prior to this upload and switch to .zst compressed firmware files. As this will be the first upload of initramfs-tools that supports .zst compressed firmware. -- okurrr, Dimitri -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Fri, 21 Jul 2023 at 21:08, Steve Langasek wrote: > > On Fri, Jul 21, 2023 at 07:33:58PM +0100, Dimitri John Ledkov wrote: > > If it is a concern that v5.15 jammy kernel may potentially be used > > after partial / incomplete upgrade to Mantic, we can opt into using XZ > > compressed firmware or I can backport ZSTD firmware support to jammy > > GA kernel - it is a trivial patch (given that ZSTD itself is otherwise > > available and used by the jammy kernel). > > The scenario here is that the user has just upgraded to mantic, so they get > the new linux firmware and the new kernel; they reboot to mantic, but there > is a regression of some kind; they reboot back to the jammy kernel (which is > the N-1 kernel on their system) to debug it, and that kernel has regressed > support for their hardware because it can no longer load the needed firmware > from the root filesystem. > > This is a realistic enough scenario that I think it is worth backporting the > zstd support onto jammy's 5.15 kernel. This portion will be tracked in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028550 submitted in https://lists.ubuntu.com/archives/kernel-team/2023-July/141355.html As previously described it was fairly trivial backport of 3 files changed, 98 insertions(+), 8 deletions(-) -- okurrr, Dimitri -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Mon, 24 Jul 2023 at 13:34, Adrien Nader wrote: > > On Tue, Jul 11, 2023, Michael Hudson-Doyle wrote: > > I was wondering if it make sense to construct a zstd dictionary for > > compressing kernel modules but I didn't realize they need to be available > > at decompression time, I'm not sure the kernel would support that. > > I've tried to use the zstd --train and I don't think it is appropriate > here, or at least not without significant preparation of datasets. It's > more suited for many many small and similar files. We don't have that > many files and there are only clusters of similarities, not similarities > shared across every file. Moreover, since our files are big (at least > the one that matter for the overall size), they're already contributing > a lot to the live dictionary and the pre-built one has little overall > influence. Also I don't think it will work for us - the train command produces dictionary that needs to be present for both compression and decompression, and right now the kernel doesn't have support for that or more specifically to load one from userspace (potentially has security implications). We could in theory generate training files, generate dictionaries, bake it into our kernel, but then it would make all of our compressed things non-portable or more difficult to update dictionary, and the .zst file itself becomes non-portable as it now depends on an external dictionary. Dictionary make sense for lots of similar things (lots of status icons, or similar gaming images) and then one can take all assets, train on them, and do a static build that uses said perfect dictionary against a perfect set of fixed assets. It sort of like the more generalised usecase of perfect-hash functions (gperf) but for data files. -- okurrr, Dimitri -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Tue, Jul 11, 2023 at 01:17:27AM +0200, Benjamin Drung wrote: > If you want to test it yourself, you can find initramfs-tools > 0.142ubuntu7bd2 for mantic in my PPA: > https://launchpad.net/~bdrung/+archive/ubuntu/ppa What blocks this from being uploaded to mantic? AFAICS the behavior should remain unchanged for existing kernel+firmware packages, and it's therefore safe to push more widely. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Fri, Jul 21, 2023 at 07:33:58PM +0100, Dimitri John Ledkov wrote: > If it is a concern that v5.15 jammy kernel may potentially be used > after partial / incomplete upgrade to Mantic, we can opt into using XZ > compressed firmware or I can backport ZSTD firmware support to jammy > GA kernel - it is a trivial patch (given that ZSTD itself is otherwise > available and used by the jammy kernel). The scenario here is that the user has just upgraded to mantic, so they get the new linux firmware and the new kernel; they reboot to mantic, but there is a regression of some kind; they reboot back to the jammy kernel (which is the N-1 kernel on their system) to debug it, and that kernel has regressed support for their hardware because it can no longer load the needed firmware from the root filesystem. This is a realistic enough scenario that I think it is worth backporting the zstd support onto jammy's 5.15 kernel. Thanks, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Fri, 21 Jul 2023 at 17:59, Steve Langasek wrote:
>
> On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote:
> > > > Okay. It works now. The not-compressed cpio archive must not be the last
> > > > one. So the order is now:
> > > >
> > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > * compressed kernel modules / firmware (not compressed)
> > > > * main cpio archive (compressed)
> > > >
> > > > I'll really stop now. For a first comparison, the firmware files need to
> > > > be converted correctly. There are symlinks in /lib/firmware. So running
> > > > following was not correct/enough:
> > > >
> > > > find /lib/firmware -name '*.bin' | while read -r fw; do
> > > > sudo zstd -19 -z -o "${fw}.zst" "$fw"
> > > > sudo rm "$fw"
> > > > done
> > > >
> > > > If you want to help, hand me a correct conversion script.
>
> > > Some filenames in /lib/firmware contain spaces. There are many more
> > > files than .bin ones. A number of the files are readmes.
>
> > > Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
> > > targets to make than will do what you want I guess. I haven't checked
> > > that this was actually merged; it was discussed at least on 2023-03-01
> > > on the mailing-list. It's probably the best path forward in any case.
>
> > > In case it isn't merged, you can have a look at
> > > https://lore.kernel.org/linux-firmware/20230301-fixes-and-compression-v2-0-e2b71974e...@gmail.com/T/
>
> > I agree. The conversion script was just for a quick way to test. The
> > clean approach would be to patch linux-firmware to produce two
> > additional binary packages: linux-firmware-zst and linux-firmware-xz.
>
> We need to find an upgrade path here that doesn't involve having to have
> multiple linux-firmware binary packages with different compression types.
> Ending up with three copies of the firmware in /lib on disk because of
> different dependencies is not an improvement!
>
> I understand the reason for being concerned about keeping uncompressed
> firmware available is that not all kernels have support for compressed
> firmware. However we should work out a path that lets us switch to
> compressed firmware on releases where we know it's supported. What kernel
> version introduced support for this?
>
XZ compressed firmware load was added in v5.3 and is turned on in the
focal GA v5.4 kernel.
ZSTD compressed firmware load was added in v5.19 and is available in
Jammy HWE kernels, kinetic and later.
initramfs-tools in Jammy has unoptimized support for xz compressed firmware.
Thus it is safe for linux-firmware to switch to ZSTD compressed
firmware straight away in mantic.
If it is a concern that v5.15 jammy kernel may potentially be used
after partial / incomplete upgrade to Mantic, we can opt into using XZ
compressed firmware or I can backport ZSTD firmware support to jammy
GA kernel - it is a trivial patch (given that ZSTD itself is otherwise
available and used by the jammy kernel).
--
okurrr,
Dimitri
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote:
> > > Okay. It works now. The not-compressed cpio archive must not be the last
> > > one. So the order is now:
> > >
> > > * AMD/Intel microcode cpio archive (on amd64)
> > > * compressed kernel modules / firmware (not compressed)
> > > * main cpio archive (compressed)
> > >
> > > I'll really stop now. For a first comparison, the firmware files need to
> > > be converted correctly. There are symlinks in /lib/firmware. So running
> > > following was not correct/enough:
> > >
> > > find /lib/firmware -name '*.bin' | while read -r fw; do
> > > sudo zstd -19 -z -o "${fw}.zst" "$fw"
> > > sudo rm "$fw"
> > > done
> > >
> > > If you want to help, hand me a correct conversion script.
> > Some filenames in /lib/firmware contain spaces. There are many more
> > files than .bin ones. A number of the files are readmes.
> > Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
> > targets to make than will do what you want I guess. I haven't checked
> > that this was actually merged; it was discussed at least on 2023-03-01
> > on the mailing-list. It's probably the best path forward in any case.
> > In case it isn't merged, you can have a look at
> > https://lore.kernel.org/linux-firmware/20230301-fixes-and-compression-v2-0-e2b71974e...@gmail.com/T/
> I agree. The conversion script was just for a quick way to test. The
> clean approach would be to patch linux-firmware to produce two
> additional binary packages: linux-firmware-zst and linux-firmware-xz.
We need to find an upgrade path here that doesn't involve having to have
multiple linux-firmware binary packages with different compression types.
Ending up with three copies of the firmware in /lib on disk because of
different dependencies is not an improvement!
I understand the reason for being concerned about keeping uncompressed
firmware available is that not all kernels have support for compressed
firmware. However we should work out a path that lets us switch to
compressed firmware on releases where we know it's supported. What kernel
version introduced support for this?
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On 13/07/2023 22:36, Benjamin Drung wrote:
One small change that brings major benefits is only including the
firmware files required, not every file declared by a kernel module. The
GPU drivers are the main culprit there with amdgpu declaring over 500 of
which, for any individual GPU, only a handful are relevant, but
Plymouth's pulling in multiple GPU drivers doesn't help that.
Statistics (initrd.img with kernel v6.2.4):
MODULES= FIRMWARE_LOADED size MOST DEP firmwares build-time
most false 77117694 634 14.49
most true60302859 -22%8 11.99
dep false 42489938 -45% 606 6.84
dep true25704125 -66% -40% 8 6.35
FIMWARE_LOADED=true relies on a simple kernel patch which I've been
meaning to upstream that writes "Firmware loaded: " for each.
$ journalctl --dmesg | grep 'Firmware loaded' | head -n 15
Jul 02 11:42:21 sunny kernel: firmware_class: Firmware loaded: reporting
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_pfp.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_me.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_ce.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_rlc.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_mc.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/verde_smc.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/TAHITI_uvd.bin
Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
radeon/TAHITI_vce.bin
Jul 02 11:42:22 sunny kernel: r8152 2-8.3:1.0: Firmware loaded:
rtl_nic/rtl8153a-4.fw
Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded:
regulatory.db
Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded:
regulatory.db.p7s
My experiments and patches are documented at
https://iam.tj/projects/ubuntu/initramfs-tools/
That is a really good idea. After looking at the patches, the options 1
and 2 (sysfs or procfs interface) are the more robust ones (but need
more work implementing them in the kernel). Option 3 (kernel logging)
seem to be too fragile for using it programmatically. What happens if a
user boot with quiet option and clears the kernel log buffer?
Thanks for looking at it. I agree about the most bullet-proof approach
but I was looking for a simple proof-of-concept at the time and then
found it so reliable I stuck with it since it is the least invasive and
probably has more chance of being accepted upstream.
As Dimitri mentions in a later response the logs from journald don't
rely on the kernel ring buffer, but as I work with both Debian and
Ubuntu I wanted to at least give a nod to non-systemd init systems hence
the fallback to dmesg.
Another idea I tested would address this; to generate and save the
currently loaded firmware list on each boot as:
/var/lib/firmware/${DMI_HASH}/${KERNEL_VERSION}.firmware
where $DMI_HASH is generated with:
cat /sys/class/dmi/id/board_{vendor,name} /sys/class/dmi/id/bios_version
| shasum
with the intention being to match identical baseboard.
The purpose behind these static files was to avoid needing to search the
kernel log every time mkinitramfs is used - when FIRMWARE_LOADED=true
this file - if it exists - would be used instead.
Remember this is a PoC that has been remarkably stable for 4 years, but
there are obvious additions to armour it against other workflows,
unusual, and corner-cases.
> Is introducing FIRMWARE_LOADED really needed? Isn't it enough to only
> include the loaded firmware for MODULES=dep and all firmware otherwise?
I'd argue it is needed. With MODULES=most there is a 22% decrease in
initrd.img size and 17% decrease in mkinitramfs time.
On systems I control there is a small separate LUKS-protected /boot/ so
any reduction in size is welcome.
$ df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/LUKS2_BOOT 461M 268M 165M 62% /boot
$ stat -c '%s %n' /boot/initrd.img-*
29404897 /boot/initrd.img-6.1.0-10-amd64
24851792 /boot/initrd.img-6.2.11-tj+
30102501 /boot/initrd.img-6.3.12+debian+tj+
28428121 /boot/initrd.img-6.3.4-debian+tj+
30179222 /boot/initrd.img-6.4.0+debian+tj+
30174161 /boot/initrd.img-6.4.2+debian+tj+
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On 13/07/2023 23:30, Dimitri John Ledkov wrote:
journald stores a copy of dmesg subject to its own retention and rotation
policy, even when dmesg is cleared. Thus it should be reliable to have
journal output for dmesg for multiple boots.
The issue however is that drivers and kernels can change on disk, and the
next kernel ABI may need different or more firmware files. examples:
1) on desktop installs hwe-22.04 rolls major versions.
2) each Nvidia SRU update needs a new firmware file name
I did do some basic investigation on this issue and played with a few
ideas; trying to find something simple.
At the time I considered the most reliable would be a file
"firmware.ids" along the lines of pci.ids/usb.ids included in each
kernel image binary package that links hardware ID to firmware as in:
firmware-file # comment
PCI # bus name
Vendor
Device1
Device2
USB
idVendor
idProduct1
idProduct2
idProduct3
firmware-file # comment
...
Being able to create it at package build-time would have some advantages
but I haven't given much thought to potential problems including
build-time complications.
The challenge for "firmware.ids" is there is (currently) no automated
way to generate it and would require either or both of kernel driver or
linux-firmware to provide a machine-readable entry. The best I could
come up with at the time was an additional line in linux-firmware WHENCE
possibly of the form:
alias: Vendor:Device Vendor:Device ...
But that has a lot of issues, not least being that the list would need
to be comprehensive and rely on human knowledge. A combination of what
is loaded plus WHENCE:alias might work though.
The list of firmware files in a kernel module would be much more
difficult to link to hardware IDs even though that is the real source of
truth since there is no declarative link between them.
There is a sneaky way to side-step the need for a list: use the
currently loaded list PLUS any firmware files newly declared by an
included (in initrd.img) module that the current module doesn't declare.
That would mean an additional call to modinfo for each included kernel
module and a diff which could be generated at kernel-install time via a
script in /etc/kernel/install.d/ that generates the
/var/lib/firmware/${DMI_HASH}/${KERNEL_VERSION}.firmware
described in my reply to Benjamin.
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Thu, 13 Jul 2023, 22:37 Benjamin Drung, wrote: > On Thu, 2023-07-13 at 08:32 +0100, TJ wrote: > > Is there a specific reason why the focus is on trying to shoe-horn > > everything possible into the initrd.img and then compress rather than > > winnow out the files an installed system will never need to find the > > root file-system? > > These are two orthogonal targets. MODULES=most is the default, because > you can change the hardware and the system will still boot. Advanced > users can use MODULES=dep to only include the needed bits, but then they > have to be aware of it when changing the hardware. Improving the way we > compress the initramfs could help in both configurations. > > > I tackled the issue of ever-expanding host-generated initrd.img file > > sizes when they began hitting 80MB+ back in 2018 and since then have > > carried my own patches that reduce sizes by not including files the host > > will not need. > > > > One small change that brings major benefits is only including the > > firmware files required, not every file declared by a kernel module. The > > GPU drivers are the main culprit there with amdgpu declaring over 500 of > > which, for any individual GPU, only a handful are relevant, but > > Plymouth's pulling in multiple GPU drivers doesn't help that. > > > > Statistics (initrd.img with kernel v6.2.4): > > MODULES= FIRMWARE_LOADED size MOST DEP firmwares build-time > > most false 77117694 634 14.49 > > most true60302859 -22%8 11.99 > > dep false 42489938 -45% 606 6.84 > > dep true25704125 -66% -40% 8 6.35 > > > > FIMWARE_LOADED=true relies on a simple kernel patch which I've been > > meaning to upstream that writes "Firmware loaded: " for each. > > > > $ journalctl --dmesg | grep 'Firmware loaded' | head -n 15 > > Jul 02 11:42:21 sunny kernel: firmware_class: Firmware loaded: reporting > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_pfp.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_me.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_ce.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_rlc.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_mc.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/verde_smc.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/TAHITI_uvd.bin > > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > > radeon/TAHITI_vce.bin > > Jul 02 11:42:22 sunny kernel: r8152 2-8.3:1.0: Firmware loaded: > > rtl_nic/rtl8153a-4.fw > > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: > > regulatory.db > > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: > > regulatory.db.p7s > > > > My experiments and patches are documented at > > > > https://iam.tj/projects/ubuntu/initramfs-tools/ > > That is a really good idea. After looking at the patches, the options 1 > and 2 (sysfs or procfs interface) are the more robust ones (but need > more work implementing them in the kernel). Option 3 (kernel logging) > seem to be too fragile for using it programmatically. What happens if a > user boot with quiet option and clears the kernel log buffer? > journald stores a copy of dmesg subject to its own retention and rotation policy, even when dmesg is cleared. Thus it should be reliable to have journal output for dmesg for multiple boots. The issue however is that drivers and kernels can change on disk, and the next kernel ABI may need different or more firmware files. examples: 1) on desktop installs hwe-22.04 rolls major versions. 2) each Nvidia SRU update needs a new firmware file name > Is introducing FIRMWARE_LOADED really needed? Isn't it enough to only > include the loaded firmware for MODULES=dep and all firmware otherwise? > > > The main take is the question: > > > > Is MODULES=most really the best default for installed-on-host images (it > > obviously is for installer and portable images). If it can be nuanced > > then =DEP plus winnowing the firmware files gives great gains with > > minimal effort. > > MODULES=most is the best default since it allows changing the underlying > hardware. > > -- > Benjamin Drung > Debian & Ubuntu Developer > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Thu, 2023-07-13 at 08:32 +0100, TJ wrote: > Is there a specific reason why the focus is on trying to shoe-horn > everything possible into the initrd.img and then compress rather than > winnow out the files an installed system will never need to find the > root file-system? These are two orthogonal targets. MODULES=most is the default, because you can change the hardware and the system will still boot. Advanced users can use MODULES=dep to only include the needed bits, but then they have to be aware of it when changing the hardware. Improving the way we compress the initramfs could help in both configurations. > I tackled the issue of ever-expanding host-generated initrd.img file > sizes when they began hitting 80MB+ back in 2018 and since then have > carried my own patches that reduce sizes by not including files the host > will not need. > > One small change that brings major benefits is only including the > firmware files required, not every file declared by a kernel module. The > GPU drivers are the main culprit there with amdgpu declaring over 500 of > which, for any individual GPU, only a handful are relevant, but > Plymouth's pulling in multiple GPU drivers doesn't help that. > > Statistics (initrd.img with kernel v6.2.4): > MODULES= FIRMWARE_LOADED size MOST DEP firmwares build-time > most false 77117694 634 14.49 > most true60302859 -22%8 11.99 > dep false 42489938 -45% 606 6.84 > dep true25704125 -66% -40% 8 6.35 > > FIMWARE_LOADED=true relies on a simple kernel patch which I've been > meaning to upstream that writes "Firmware loaded: " for each. > > $ journalctl --dmesg | grep 'Firmware loaded' | head -n 15 > Jul 02 11:42:21 sunny kernel: firmware_class: Firmware loaded: reporting > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_pfp.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_me.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_ce.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_rlc.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_mc.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/verde_smc.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/TAHITI_uvd.bin > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: > radeon/TAHITI_vce.bin > Jul 02 11:42:22 sunny kernel: r8152 2-8.3:1.0: Firmware loaded: > rtl_nic/rtl8153a-4.fw > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: > regulatory.db > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: > regulatory.db.p7s > > My experiments and patches are documented at > > https://iam.tj/projects/ubuntu/initramfs-tools/ That is a really good idea. After looking at the patches, the options 1 and 2 (sysfs or procfs interface) are the more robust ones (but need more work implementing them in the kernel). Option 3 (kernel logging) seem to be too fragile for using it programmatically. What happens if a user boot with quiet option and clears the kernel log buffer? Is introducing FIRMWARE_LOADED really needed? Isn't it enough to only include the loaded firmware for MODULES=dep and all firmware otherwise? > The main take is the question: > > Is MODULES=most really the best default for installed-on-host images (it > obviously is for installer and portable images). If it can be nuanced > then =DEP plus winnowing the firmware files gives great gains with > minimal effort. MODULES=most is the best default since it allows changing the underlying hardware. -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Thu, Jul 13, 2023, Julian Andres Klode wrote: > On Thu, Jul 13, 2023 at 08:32:48AM +0100, TJ wrote: > > Is there a specific reason why the focus is on trying to shoe-horn > > everything possible into the initrd.img and then compress rather than winnow > > out the files an installed system will never need to find the root > > file-system? > > Basically the rule is that if you unplug the disk from your system and > plug it into another it should still boot. > > You can also consider that an upgrade problem for PCs. When you replace > the mainboard with a different one, the install should continue working. I should point out that I have a tool that can alleviate the issue of duplication for amdgpu firmware files. It's almost ready; it just needs some polish and integration (it requires a very slightly patched xz encoder which I wanted to avoid but I've given up on improving that without additional help). Basically you don't care much that you have firmware for 20 GPU families since the firmware files are very similar and will compress very well provided they appear in an acceptable order in the archive. I'll try to get that ready for everyone to use in the coming weeks. -- Adrien -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Thu, Jul 13, 2023 at 08:32:48AM +0100, TJ wrote: > Is there a specific reason why the focus is on trying to shoe-horn > everything possible into the initrd.img and then compress rather than winnow > out the files an installed system will never need to find the root > file-system? Basically the rule is that if you unplug the disk from your system and plug it into another it should still boot. You can also consider that an upgrade problem for PCs. When you replace the mainboard with a different one, the install should continue working. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
Is there a specific reason why the focus is on trying to shoe-horn everything possible into the initrd.img and then compress rather than winnow out the files an installed system will never need to find the root file-system? I tackled the issue of ever-expanding host-generated initrd.img file sizes when they began hitting 80MB+ back in 2018 and since then have carried my own patches that reduce sizes by not including files the host will not need. One small change that brings major benefits is only including the firmware files required, not every file declared by a kernel module. The GPU drivers are the main culprit there with amdgpu declaring over 500 of which, for any individual GPU, only a handful are relevant, but Plymouth's pulling in multiple GPU drivers doesn't help that. Statistics (initrd.img with kernel v6.2.4): MODULES= FIRMWARE_LOADED size MOST DEP firmwares build-time most false 77117694 634 14.49 most true60302859 -22%8 11.99 dep false 42489938 -45% 606 6.84 dep true25704125 -66% -40% 8 6.35 FIMWARE_LOADED=true relies on a simple kernel patch which I've been meaning to upstream that writes "Firmware loaded: " for each. $ journalctl --dmesg | grep 'Firmware loaded' | head -n 15 Jul 02 11:42:21 sunny kernel: firmware_class: Firmware loaded: reporting Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_pfp.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_me.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_ce.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_rlc.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_mc.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/verde_smc.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/TAHITI_uvd.bin Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded: radeon/TAHITI_vce.bin Jul 02 11:42:22 sunny kernel: r8152 2-8.3:1.0: Firmware loaded: rtl_nic/rtl8153a-4.fw Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: regulatory.db Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded: regulatory.db.p7s My experiments and patches are documented at https://iam.tj/projects/ubuntu/initramfs-tools/ The main take is the question: Is MODULES=most really the best default for installed-on-host images (it obviously is for installer and portable images). If it can be nuanced then =DEP plus winnowing the firmware files gives great gains with minimal effort. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Tue, Jul 11, 2023, Seth Arnold wrote: > On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > > There is a little-know but very interesting property of LZMA: its > > decompression speed does not depend on the uncompressed size but only on > > the compressed size. What this means is that if you compress a 100MB > > down to 20MB, it will take roughly twice as long to decompress than if > > you compress it down to 10MB. In other words, higher compression means > > faster decompression. > > This makes a certain amount of sense -- so much of a computer's > operational time is spent waiting for data to arrive from memory into > the processor, refilling cache lines, etc. > > You nerd-sniped me into testing a bunch of algorithms on the > firefox_115.0+build2.orig.tar from our archive. > > I only ran these things once, and quite a lot of them ran while a > second one was running, but this system (dual xeon E5-2630v3) has enough > processors and memory that it probably didn't matter much. > > Times in seconds, with lower level on the left, higher on the right: > > 1 3 5 9 > compression: > gzip39 46 73 211 > zstd 8 12 23 54 > bzip2 228 237 249 265 > lzma 154 294 643 945 > xz 159 298 644 945 > > decompression: > gzip16 15 15 15 > zstd 3 3 3 3 > bzip2 68 73 74 75 > lzma41 37 35 33 > xz 36 32 31 30 > > xz of course absolutely dominates the end file sizes: > > 2989486080 original > > 515273416 xz -9 > 625958113 zstd -9 > 647365812 xz -1 > 666820870 zstd -5 (seemed like a sweet spot in the timings) > > Anyway it's fun to see that gzip and zstd have consistent decompression > speeds, lzma and xz get faster as they go smaller, and bzip2 just gets > slower the more it has to think. There's one more dimension to explore and since it combines with other settins, it makes things quite a bit more difficult. With xz you can easily tweak dictionary size (it's roughly equivalent to a "window size"): xz -v --lzma2=preset=0,dict=1536M -k firefox_115.0+build2.orig.tar Two comments: a) xz has a -0 preset, b) 1536M is the maximum dictionary size for LZMA2. Zstd has --zstd=windowLog=n where the window size is 2^n. Memory usage during compression can reach at least 2*2^n; since n<=30, that's <= 2GB. I'm not sure about the usage during decompression. Some more data points (obviously CPU times are different): CompressorOutput size Comp. time Decomp. time --- xz -0 704087984 92s 27s xz -0,dict=1.5G 617397528 168s 22s xz -1 647365812 118s 24s xz -9 515273416 753s 20s xz -9,dict=1.5G 485393536 946s 19s zstd -1 764765167 5s <3s zstd -5 666820870 17s <3s zstd -5,win=18 712243629 18s <3s zstd -5,win=19 694314044 18s <3s zstd -9 625958113 34s <3s zstd -9,win=30 588361999 61s <3s zstd -19,win=30 5092519041238s <3s The presets for xz and zstd are very general-purpose. They're meant to gradually provider better compression at the expense of CPU and memory usage. I'm sure we can tweak them and achieve very interesting results. At the scale of Ubuntu, we might want to deviate from presets simply because we have a good enough idea of which hardware runs Ubuntu. I know that the memory usage of xz during decompression was a hot topic ... 15 years ago: NetBSD people had issues on some machine that didn't have 64MB of memory available. We're past that except for maybe some very specific hardware. For specific hardware with low memory we might also want to deviate from presets but doing the opposite. In my tests above, zstd -5 with windowLog=18 users 13MB of memory while with windowLog=19 it uses 20MB; zstd -1 uses 16MB. All this is different from the initial topic but I really believe there's a lot of potential there for everything we compress. -- Adrien -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Tue, Jul 11, 2023 at 12:53:00AM +, Seth Arnold wrote: > On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > > There is a little-know but very interesting property of LZMA: its > > decompression speed does not depend on the uncompressed size but only on > > the compressed size. What this means is that if you compress a 100MB > > down to 20MB, it will take roughly twice as long to decompress than if > > you compress it down to 10MB. In other words, higher compression means > > faster decompression. > > This makes a certain amount of sense -- so much of a computer's > operational time is spent waiting for data to arrive from memory into > the processor, refilling cache lines, etc. > > You nerd-sniped me into testing a bunch of algorithms on the > firefox_115.0+build2.orig.tar from our archive. > > I only ran these things once, and quite a lot of them ran while a > second one was running, but this system (dual xeon E5-2630v3) has enough > processors and memory that it probably didn't matter much. > > Times in seconds, with lower level on the left, higher on the right: > > 1 3 5 9 > compression: > gzip39 46 73 211 > zstd 8 12 23 54 > bzip2 228 237 249 265 > lzma 154 294 643 945 > xz 159 298 644 945 > > decompression: > gzip16 15 15 15 > zstd 3 3 3 3 > bzip2 68 73 74 75 > lzma41 37 35 33 > xz 36 32 31 30 > > xz of course absolutely dominates the end file sizes: > > 2989486080 original > > 515273416 xz -9 > 625958113 zstd -9 > 647365812 xz -1 > 666820870 zstd -5 (seemed like a sweet spot in the timings) > > Anyway it's fun to see that gzip and zstd have consistent decompression > speeds, lzma and xz get faster as they go smaller, and bzip2 just gets > slower the more it has to think. One caveat to your whole experiment is that zstd -9 is roughly comparable to gzip -9, whereas zstd -19 is the level comparable to xz -9 (arguably you may also pick 17, 18 depending on how long the compression takes vs xz -9 :D). I think that's a reason you don't see speedups because zstd is already quite fast and you are not really using high compression levels. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Tue, 11 Jul 2023 at 11:17, Benjamin Drung wrote: > On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote: > > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, > 02:19: > > > > > > > > Hi all, > > > > > > > > a year ago we changed the default compression and level for the > > > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > > > > development boards (see bug #1958148), but that leads to bigger > > > > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > > > > Big initramfs sizes can also fill up small sized /boot > partitions easily > > > > > > (grooming the 850 initramfs-tools bugs revealed several such > reports). > > > > > > > > Using xz -9 would give very good compression, but it takes very > long > > > > > > (especially on slow development boards) and a lot of memory > (good luck > > > > > > on Raspberry Pis with small memory like Pi Zeros). > > > > > > > > I propose following approach to address the drawback: Create cpio > > > > > > archives (compressed with xz -9) for the kernel modules and > firmware > > > > > > files when building the kernel/firmware Debian package. Then > ship those > > > > > > cpio archives in the package (or in a separate binary package). > Then the > > > > > > CPU load it put on the builders. The cpio archives would contain > the > > > > > > modules for MODULES=most. > > > > > > > > mkinitramfs will then look for those cpio archives and uses > those in > > > > > > case they are present. Such a initramfs would look like this: > > > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > > > > * main cpio archive compressed with zstd -1 > > > > > > * kernel modules from the Debian package compressed with xz -9 > > > > > > * firmware files from the Debian package compressed with xz -9 > > > > > > > > After working on initramfs-tools as part my day job, my fingers > were > > > > > > itching and I had to create a quick and dirty draft in my free > night > > > > > > time. You can find the result of the last two hours in [1]. This > draft > > > > > > has a mkinitramfs-kernel script that creates a cpio archive > containing > > > > > > the kernel modules and firmware (that needs to be split later > on). > > > > > > > > The lunar test result on my AMD Ryzen 7 5700G look promising: > Building > > > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds > and is > > > > > > 54.9 MiB in size. Creating the initramfs speeds up from around > 8.7 > > > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 > MiB to > > > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > > > > > > /lib/modules need 54.9 MiB for the cpio archive. > > > > > > > > The drawback is that building the kernel would take longer, the > package > > > > > > takes more space on the archive and mirrors, and downloading > them could > > > > > > take longer on slow connections. > > > > > > > > Implementing my proposal would be relative easy for > initramfs-tools, but > > > > > > would mean some work for the kernel team. > > > > > > > > What do you think? > > > > > Will the user still be able to add further modules and will machine > specific > > > configuration files (e.g. for booting from iSCSI) still be included > into the > > > initrd? > > > > I think a robust implementation of this on the initramfs-tools side looks > > like: > > > > - identify all the contents that belong in the initramfs > > - among those contents, find all zstd-compressed files, if any, and > store > >them in an uncompressed initramfs > > - put the rest of the contents in a compressed initramfs > > > > This would be compatible with kernel packages whether they are changed to > > ship zstd-compressed modules or not and allow for a smooth transition. > > This is exactly what I tried to implement over the weekend. The result > looked promising. The creation time and initramfs size decreased by a > few percent. Booting that initramfs failed and today I figured out why. > Now I have a new draft version that works, but is slower to compress, > because it moves the compressed files around after running depmod. > Fiddling with the list of files passed to cpio should give back the > speed, but that is a task for later. > > So without further ado here is the result so far: > These look like excellent results, and this approach makes a lot of sense to me. I've wanted to see this improved for ages, so I'm happy to see someone actually taking steps to make this happen! Spending builder time on maximum compression vs doing it on every Ubuntu system in existence is surely better for everyone. Would the idea be that linux-modules-$VER-generic would contain only the compressed modules? I was wondering if it make sense to construct a zstd dictionary for compressing kernel modules but I didn't realize they need to be available at decompression time, I'm not sure the kernel would
Re: Reducing initramfs size and speed up the generation
On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > There is a little-know but very interesting property of LZMA: its > decompression speed does not depend on the uncompressed size but only on > the compressed size. What this means is that if you compress a 100MB > down to 20MB, it will take roughly twice as long to decompress than if > you compress it down to 10MB. In other words, higher compression means > faster decompression. This makes a certain amount of sense -- so much of a computer's operational time is spent waiting for data to arrive from memory into the processor, refilling cache lines, etc. You nerd-sniped me into testing a bunch of algorithms on the firefox_115.0+build2.orig.tar from our archive. I only ran these things once, and quite a lot of them ran while a second one was running, but this system (dual xeon E5-2630v3) has enough processors and memory that it probably didn't matter much. Times in seconds, with lower level on the left, higher on the right: 1 3 5 9 compression: gzip39 46 73 211 zstd 8 12 23 54 bzip2 228 237 249 265 lzma 154 294 643 945 xz 159 298 644 945 decompression: gzip16 15 15 15 zstd 3 3 3 3 bzip2 68 73 74 75 lzma41 37 35 33 xz 36 32 31 30 xz of course absolutely dominates the end file sizes: 2989486080 original 515273416 xz -9 625958113 zstd -9 647365812 xz -1 666820870 zstd -5 (seemed like a sweet spot in the timings) Anyway it's fun to see that gzip and zstd have consistent decompression speeds, lzma and xz get faster as they go smaller, and bzip2 just gets slower the more it has to think. Thanks signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Mon, 2023-07-10 at 11:09 +0200, Adrien Nader wrote:
> On Sat, Jul 08, 2023, Benjamin Drung wrote:
> > On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote:
> > > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> > > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > a year ago we changed the default compression and level for the
> > > > > initramfs to zstd -1. This fixed the very slow creation times on
> > > > > development boards (see bug #1958148), but that leads to bigger
> > > > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > > > Big initramfs sizes can also fill up small sized /boot partitions
> > > > > easily
> > > > > (grooming the 850 initramfs-tools bugs revealed several such reports).
> > > > >
> > > > > Using xz -9 would give very good compression, but it takes very long
> > > > > (especially on slow development boards) and a lot of memory (good luck
> > > > > on Raspberry Pis with small memory like Pi Zeros).
> > > > >
> > > > > I propose following approach to address the drawback: Create cpio
> > > > > archives (compressed with xz -9) for the kernel modules and firmware
> > > > > files when building the kernel/firmware Debian package. Then ship
> > > > > those
> > > > > cpio archives in the package (or in a separate binary package). Then
> > > > > the
> > > > > CPU load it put on the builders. The cpio archives would contain the
> > > > > modules for MODULES=most.
> > > > >
> > > > > mkinitramfs will then look for those cpio archives and uses those in
> > > > > case they are present. Such a initramfs would look like this:
> > > > >
> > > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > > * main cpio archive compressed with zstd -1
> > > > > * kernel modules from the Debian package compressed with xz -9
> > > > > * firmware files from the Debian package compressed with xz -9
> > > > >
> > > >
> > > > Majority of our instances boot without initrd, and there too they
> > > > don't load most of the modules.
> > > > Creating xz -9 compressed archive of all modules, still pays the
> > > > penalty to decompress most of them, and then not modprobe them.
> > > > I was hoping to achieve a similar in spirit approach, but didn't quite
> > > > have the time to implement is:
> > > >
> > > > 1) change linux-modules and linux-firmware to ship .ko.zst
> > > > firmware.bin.zst compressed with zstd -19 at .deb build time
> > > > 2) this saves install size of the packages, with only slightly
> > > > increased download size
> > > > 3) modify initramfs-tools to include compressed files into a separate
> > > > initrd, which is not compressed (i.e. exclude .zst files from the
> > > > default main compressed cpio archive, and append them in the second
> > > > main cpio archive that is uncompressed)
> > > > 4) this should achieve quick initrd creation, which will be smaller in
> > > > size that current status, and will boot faster as it will only
> > > > decompress modules/firmware it actually needs at boot
> > > >
> > > > For experimentation locally, you can recompress .ko with zstd in place
> > > > in /lib/modules/; and rerun depmod. To then test initramfs-tools
> > > > changes that skip over .zst compressed files and add them as is in an
> > > > uncompressed appended cpio.
> > >
> > > That is a very good idea. I created a draft for point 3 in [2]. It moves
> > > the compressed files into a separate directory and creates a separate
> > > cpio archive for that directory without compressing it:
> > >
> > > * AMD/Intel microcode cpio archive (on amd64)
> > > * main cpio archive (compressed)
> > > * compressed kernel modules / firmware (not compressed)
> > >
> > > Sadly this does not work (yet). cpio complains with "premature end of
> > > archive" when looking at it and the kernel fails to extract the last
> > > cpio part. I am heading to bed now leaving that bug for another day.
> > >
> > > [2]
> > > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed
> >
> > Okay. It works now. The not-compressed cpio archive must not be the last
> > one. So the order is now:
> >
> > * AMD/Intel microcode cpio archive (on amd64)
> > * compressed kernel modules / firmware (not compressed)
> > * main cpio archive (compressed)
> >
> > I'll really stop now. For a first comparison, the firmware files need to
> > be converted correctly. There are symlinks in /lib/firmware. So running
> > following was not correct/enough:
> >
> > find /lib/firmware -name '*.bin' | while read -r fw; do
> > sudo zstd -19 -z -o "${fw}.zst" "$fw"
> > sudo rm "$fw"
> > done
> >
> > If you want to help, hand me a correct conversion script.
>
> Some filenames in /lib/firmware contain spaces. There are many more
> files than .bin ones. A number of the files are readmes.
>
> Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
> targets to make than will do
Re: Reducing initramfs size and speed up the generation
On Mon, 2023-07-10 at 16:12 +0200, Heinrich Schuchardt wrote: > On 10.07.23 00:29, Steve Langasek wrote: > > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, > > > > > 02:19: > > > > > > > > Hi all, > > > > > > > > a year ago we changed the default compression and level for the > > > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > > > > development boards (see bug #1958148), but that leads to bigger > > > > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > > > > Big initramfs sizes can also fill up small sized /boot partitions > > > > > > easily > > > > > > (grooming the 850 initramfs-tools bugs revealed several such > > > > > > reports). > > > > > > > > Using xz -9 would give very good compression, but it takes very long > > > > > > (especially on slow development boards) and a lot of memory (good > > > > > > luck > > > > > > on Raspberry Pis with small memory like Pi Zeros). > > > > > > > > I propose following approach to address the drawback: Create cpio > > > > > > archives (compressed with xz -9) for the kernel modules and firmware > > > > > > files when building the kernel/firmware Debian package. Then ship > > > > > > those > > > > > > cpio archives in the package (or in a separate binary package). > > > > > > Then the > > > > > > CPU load it put on the builders. The cpio archives would contain the > > > > > > modules for MODULES=most. > > > > > > > > mkinitramfs will then look for those cpio archives and uses those in > > > > > > case they are present. Such a initramfs would look like this: > > > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > > > > * main cpio archive compressed with zstd -1 > > > > > > * kernel modules from the Debian package compressed with xz -9 > > > > > > * firmware files from the Debian package compressed with xz -9 > > > > > > > > After working on initramfs-tools as part my day job, my fingers were > > > > > > itching and I had to create a quick and dirty draft in my free night > > > > > > time. You can find the result of the last two hours in [1]. This > > > > > > draft > > > > > > has a mkinitramfs-kernel script that creates a cpio archive > > > > > > containing > > > > > > the kernel modules and firmware (that needs to be split later on). > > > > > > > > The lunar test result on my AMD Ryzen 7 5700G look promising: > > > > > > Building > > > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > > > > > > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > > > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 > > > > > > MiB to > > > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > > > > > > /lib/modules need 54.9 MiB for the cpio archive. > > > > > > > > The drawback is that building the kernel would take longer, the > > > > > > package > > > > > > takes more space on the archive and mirrors, and downloading them > > > > > > could > > > > > > take longer on slow connections. > > > > > > > > Implementing my proposal would be relative easy for > > > > > > initramfs-tools, but > > > > > > would mean some work for the kernel team. > > > > > > > > What do you think? > > > > > Will the user still be able to add further modules and will machine > > > specific > > > configuration files (e.g. for booting from iSCSI) still be included into > > > the > > > initrd? > > > > I think a robust implementation of this on the initramfs-tools side looks > > like: > > > > - identify all the contents that belong in the initramfs > > - among those contents, find all zstd-compressed files, if any, and store > > them in an uncompressed initramfs > > - put the rest of the contents in a compressed initramfs > > > > This would be compatible with kernel packages whether they are changed to > > ship zstd-compressed modules or not and allow for a smooth transition. > > > > > > I built a kernel with CONFIG_MODULE_COMPRESS_ZSTD=y. > > update-initramfs cannot find a module specified in > /etc/initramfs-tools/conf.d/modules_list.conf due to the .zstd extension. That should work with the current initramfs-tools 0.142ubuntu7. This version only ignores zstd compressed firmware file (and uncompresses xz and zst kernel modules before creating the initramfs). > Modules are uncompressed before being added to the initrd. > > We would need an updated initramfs-tools package for evaluating that path. Here you have initramfs-tools 0.142ubuntu7bd2 for mantic in my PPA https://launchpad.net/~bdrung/+archive/ubuntu/ppa for testing. -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote: > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > > > > > > Hi all, > > > > > > a year ago we changed the default compression and level for the > > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > > > development boards (see bug #1958148), but that leads to bigger > > > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > > > Big initramfs sizes can also fill up small sized /boot partitions > > > > > easily > > > > > (grooming the 850 initramfs-tools bugs revealed several such reports). > > > > > > Using xz -9 would give very good compression, but it takes very long > > > > > (especially on slow development boards) and a lot of memory (good luck > > > > > on Raspberry Pis with small memory like Pi Zeros). > > > > > > I propose following approach to address the drawback: Create cpio > > > > > archives (compressed with xz -9) for the kernel modules and firmware > > > > > files when building the kernel/firmware Debian package. Then ship > > > > > those > > > > > cpio archives in the package (or in a separate binary package). Then > > > > > the > > > > > CPU load it put on the builders. The cpio archives would contain the > > > > > modules for MODULES=most. > > > > > > mkinitramfs will then look for those cpio archives and uses those in > > > > > case they are present. Such a initramfs would look like this: > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > > > * main cpio archive compressed with zstd -1 > > > > > * kernel modules from the Debian package compressed with xz -9 > > > > > * firmware files from the Debian package compressed with xz -9 > > > > > > After working on initramfs-tools as part my day job, my fingers were > > > > > itching and I had to create a quick and dirty draft in my free night > > > > > time. You can find the result of the last two hours in [1]. This draft > > > > > has a mkinitramfs-kernel script that creates a cpio archive containing > > > > > the kernel modules and firmware (that needs to be split later on). > > > > > > The lunar test result on my AMD Ryzen 7 5700G look promising: Building > > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > > > > > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB > > > > > to > > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > > > > > /lib/modules need 54.9 MiB for the cpio archive. > > > > > > The drawback is that building the kernel would take longer, the > > > > > package > > > > > takes more space on the archive and mirrors, and downloading them > > > > > could > > > > > take longer on slow connections. > > > > > > Implementing my proposal would be relative easy for initramfs-tools, > > > > > but > > > > > would mean some work for the kernel team. > > > > > > What do you think? > > > Will the user still be able to add further modules and will machine specific > > configuration files (e.g. for booting from iSCSI) still be included into the > > initrd? > > I think a robust implementation of this on the initramfs-tools side looks > like: > > - identify all the contents that belong in the initramfs > - among those contents, find all zstd-compressed files, if any, and store >them in an uncompressed initramfs > - put the rest of the contents in a compressed initramfs > > This would be compatible with kernel packages whether they are changed to > ship zstd-compressed modules or not and allow for a smooth transition. This is exactly what I tried to implement over the weekend. The result looked promising. The creation time and initramfs size decreased by a few percent. Booting that initramfs failed and today I figured out why. Now I have a new draft version that works, but is slower to compress, because it moves the compressed files around after running depmod. Fiddling with the list of files passed to cpio should give back the speed, but that is a task for later. So without further ado here is the result so far: | test| size / b | uncompressed / b | |-||--| | dt original | 139679329 | 481805575| | dt zstd -19 | 133992932 | 195432237| |-||--| | vm original | 68421993 | 229445821| | vm zstd -19 | 62147141 | 111007617| | vm xz -9| 58592562 | 107451029| or converted to MiB: | test| size / MiB | uncomp. / MiB | |-||---| | dt original | 133.2 | 459.4| | dt zstd -19 | 127.8 | 186.3| |-||---| | vm original | 65.3 | 218.8| | vm zstd -19 | 59.3 | 105.9| | vm xz -9| 55.9 | 102.5| Legend:
Re: Reducing initramfs size and speed up the generation
On Sat, Jul 08, 2023, Seth Arnold wrote:
> On Sat, Jul 08, 2023 at 02:18:52AM +0200, Benjamin Drung wrote:
> > Using xz -9 would give very good compression, but it takes very long
> > (especially on slow development boards) and a lot of memory (good luck
> > on Raspberry Pis with small memory like Pi Zeros).
>
> Please also test decompression times for the various algorithms. I have
> a special dislike for xz after seeing how long it takes even decent
> machines to unpack Debian packages that were compressed with xz.
>
> It's hard to know how exactly to balance compress and decompress times,
> different users will have different ratios of booting vs installing
> updates.
There is a little-know but very interesting property of LZMA: its
decompression speed does not depend on the uncompressed size but only on
the compressed size. What this means is that if you compress a 100MB
down to 20MB, it will take roughly twice as long to decompress than if
you compress it down to 10MB. In other words, higher compression means
faster decompression.
There are two main ways to improve compression ratio with xz: use a
large dictionary or use a higher compression level (which increases the
dictionary level too). The drawback to larger dictionary is larger
memory usage: during compression roughly 10*${dict_size} is needed and
during decompression, it is roughtly ${dict_size}. With xz -9's 64MB
dictionary, you would get ~65MB for decompression and 674MB for
compression (you can run "xz -9vv" to get that information).
As such it is worth deciding what is our max memory usage during
compression and decompression. I think we can afford to use much more
than 64MB of memory when decompressing initrds and that the builders
have enough memory too.
This might also apply to some of zstd's algorithm but I can't guarantee
it because while I'm at least somewhat familiar with xz, zstd is much
more complex and AFAIU uses different algorithms depending on the
compression level that is used.
--
Adrien
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, Jul 08, 2023, Benjamin Drung wrote:
> On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote:
> > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote:
> > > >
> > > > Hi all,
> > > >
> > > > a year ago we changed the default compression and level for the
> > > > initramfs to zstd -1. This fixed the very slow creation times on
> > > > development boards (see bug #1958148), but that leads to bigger
> > > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > > Big initramfs sizes can also fill up small sized /boot partitions easily
> > > > (grooming the 850 initramfs-tools bugs revealed several such reports).
> > > >
> > > > Using xz -9 would give very good compression, but it takes very long
> > > > (especially on slow development boards) and a lot of memory (good luck
> > > > on Raspberry Pis with small memory like Pi Zeros).
> > > >
> > > > I propose following approach to address the drawback: Create cpio
> > > > archives (compressed with xz -9) for the kernel modules and firmware
> > > > files when building the kernel/firmware Debian package. Then ship those
> > > > cpio archives in the package (or in a separate binary package). Then the
> > > > CPU load it put on the builders. The cpio archives would contain the
> > > > modules for MODULES=most.
> > > >
> > > > mkinitramfs will then look for those cpio archives and uses those in
> > > > case they are present. Such a initramfs would look like this:
> > > >
> > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > * main cpio archive compressed with zstd -1
> > > > * kernel modules from the Debian package compressed with xz -9
> > > > * firmware files from the Debian package compressed with xz -9
> > > >
> > >
> > > Majority of our instances boot without initrd, and there too they
> > > don't load most of the modules.
> > > Creating xz -9 compressed archive of all modules, still pays the
> > > penalty to decompress most of them, and then not modprobe them.
> > > I was hoping to achieve a similar in spirit approach, but didn't quite
> > > have the time to implement is:
> > >
> > > 1) change linux-modules and linux-firmware to ship .ko.zst
> > > firmware.bin.zst compressed with zstd -19 at .deb build time
> > > 2) this saves install size of the packages, with only slightly
> > > increased download size
> > > 3) modify initramfs-tools to include compressed files into a separate
> > > initrd, which is not compressed (i.e. exclude .zst files from the
> > > default main compressed cpio archive, and append them in the second
> > > main cpio archive that is uncompressed)
> > > 4) this should achieve quick initrd creation, which will be smaller in
> > > size that current status, and will boot faster as it will only
> > > decompress modules/firmware it actually needs at boot
> > >
> > > For experimentation locally, you can recompress .ko with zstd in place
> > > in /lib/modules/; and rerun depmod. To then test initramfs-tools
> > > changes that skip over .zst compressed files and add them as is in an
> > > uncompressed appended cpio.
> >
> > That is a very good idea. I created a draft for point 3 in [2]. It moves
> > the compressed files into a separate directory and creates a separate
> > cpio archive for that directory without compressing it:
> >
> > * AMD/Intel microcode cpio archive (on amd64)
> > * main cpio archive (compressed)
> > * compressed kernel modules / firmware (not compressed)
> >
> > Sadly this does not work (yet). cpio complains with "premature end of
> > archive" when looking at it and the kernel fails to extract the last
> > cpio part. I am heading to bed now leaving that bug for another day.
> >
> > [2]
> > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed
>
> Okay. It works now. The not-compressed cpio archive must not be the last
> one. So the order is now:
>
> * AMD/Intel microcode cpio archive (on amd64)
> * compressed kernel modules / firmware (not compressed)
> * main cpio archive (compressed)
>
> I'll really stop now. For a first comparison, the firmware files need to
> be converted correctly. There are symlinks in /lib/firmware. So running
> following was not correct/enough:
>
> find /lib/firmware -name '*.bin' | while read -r fw; do
> sudo zstd -19 -z -o "${fw}.zst" "$fw"
> sudo rm "$fw"
> done
>
> If you want to help, hand me a correct conversion script.
Some filenames in /lib/firmware contain spaces. There are many more
files than .bin ones. A number of the files are readmes.
Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
targets to make than will do what you want I guess. I haven't checked
that this was actually merged; it was discussed at least on 2023-03-01
on the mailing-list. It's probably the best path forward in any case.
In case it isn't merged, you can have a look at
Re: Reducing initramfs size and speed up the generation
On 10.07.23 00:29, Steve Langasek wrote: On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: Hi all, a year ago we changed the default compression and level for the initramfs to zstd -1. This fixed the very slow creation times on development boards (see bug #1958148), but that leads to bigger initramfs sizes that triggered other bugs (like bug #1842320). Big initramfs sizes can also fill up small sized /boot partitions easily (grooming the 850 initramfs-tools bugs revealed several such reports). Using xz -9 would give very good compression, but it takes very long (especially on slow development boards) and a lot of memory (good luck on Raspberry Pis with small memory like Pi Zeros). I propose following approach to address the drawback: Create cpio archives (compressed with xz -9) for the kernel modules and firmware files when building the kernel/firmware Debian package. Then ship those cpio archives in the package (or in a separate binary package). Then the CPU load it put on the builders. The cpio archives would contain the modules for MODULES=most. mkinitramfs will then look for those cpio archives and uses those in case they are present. Such a initramfs would look like this: * AMD/Intel microcode cpio archive (on amd64) * main cpio archive compressed with zstd -1 * kernel modules from the Debian package compressed with xz -9 * firmware files from the Debian package compressed with xz -9 After working on initramfs-tools as part my day job, my fingers were itching and I had to create a quick and dirty draft in my free night time. You can find the result of the last two hours in [1]. This draft has a mkinitramfs-kernel script that creates a cpio archive containing the kernel modules and firmware (that needs to be split later on). The lunar test result on my AMD Ryzen 7 5700G look promising: Building 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but /lib/modules need 54.9 MiB for the cpio archive. The drawback is that building the kernel would take longer, the package takes more space on the archive and mirrors, and downloading them could take longer on slow connections. Implementing my proposal would be relative easy for initramfs-tools, but would mean some work for the kernel team. What do you think? Will the user still be able to add further modules and will machine specific configuration files (e.g. for booting from iSCSI) still be included into the initrd? I think a robust implementation of this on the initramfs-tools side looks like: - identify all the contents that belong in the initramfs - among those contents, find all zstd-compressed files, if any, and store them in an uncompressed initramfs - put the rest of the contents in a compressed initramfs This would be compatible with kernel packages whether they are changed to ship zstd-compressed modules or not and allow for a smooth transition. I built a kernel with CONFIG_MODULE_COMPRESS_ZSTD=y. update-initramfs cannot find a module specified in /etc/initramfs-tools/conf.d/modules_list.conf due to the .zstd extension. Modules are uncompressed before being added to the initrd. We would need an updated initramfs-tools package for evaluating that path. Best regards Heinrich -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > > > > Hi all, > > > > a year ago we changed the default compression and level for the > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > > development boards (see bug #1958148), but that leads to bigger > > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > > Big initramfs sizes can also fill up small sized /boot partitions easily > > > > (grooming the 850 initramfs-tools bugs revealed several such reports). > > > > Using xz -9 would give very good compression, but it takes very long > > > > (especially on slow development boards) and a lot of memory (good luck > > > > on Raspberry Pis with small memory like Pi Zeros). > > > > I propose following approach to address the drawback: Create cpio > > > > archives (compressed with xz -9) for the kernel modules and firmware > > > > files when building the kernel/firmware Debian package. Then ship those > > > > cpio archives in the package (or in a separate binary package). Then the > > > > CPU load it put on the builders. The cpio archives would contain the > > > > modules for MODULES=most. > > > > mkinitramfs will then look for those cpio archives and uses those in > > > > case they are present. Such a initramfs would look like this: > > > > * AMD/Intel microcode cpio archive (on amd64) > > > > * main cpio archive compressed with zstd -1 > > > > * kernel modules from the Debian package compressed with xz -9 > > > > * firmware files from the Debian package compressed with xz -9 > > > > After working on initramfs-tools as part my day job, my fingers were > > > > itching and I had to create a quick and dirty draft in my free night > > > > time. You can find the result of the last two hours in [1]. This draft > > > > has a mkinitramfs-kernel script that creates a cpio archive containing > > > > the kernel modules and firmware (that needs to be split later on). > > > > The lunar test result on my AMD Ryzen 7 5700G look promising: Building > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > > > > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > > > > /lib/modules need 54.9 MiB for the cpio archive. > > > > The drawback is that building the kernel would take longer, the package > > > > takes more space on the archive and mirrors, and downloading them could > > > > take longer on slow connections. > > > > Implementing my proposal would be relative easy for initramfs-tools, but > > > > would mean some work for the kernel team. > > > > What do you think? > Will the user still be able to add further modules and will machine specific > configuration files (e.g. for booting from iSCSI) still be included into the > initrd? I think a robust implementation of this on the initramfs-tools side looks like: - identify all the contents that belong in the initramfs - among those contents, find all zstd-compressed files, if any, and store them in an uncompressed initramfs - put the rest of the contents in a compressed initramfs This would be compatible with kernel packages whether they are changed to ship zstd-compressed modules or not and allow for a smooth transition. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On 7/8/23 21:42, Steve Langasek wrote: Hi Heinrich, On Sat, Jul 08, 2023 at 10:44:58AM +0200, Heinrich Schuchardt wrote: Hello Benjamin, some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that a default initrd cannot be decompressed. Some boards may need modules that are not in initrd by default. Can you clarify where exactly this limitation is? Is this because the system must have room in memory both for the original initramfs image as it is loaded into memory from disk, regardless of compression; plus an uncompressed copy used as the VFS? If that's the case, wouldn't having the kernel modules compressed inside of an uncompressed cpio archive reduce overall memory usage, because the unpacking of the initramfs doesn't uncompress these compressed files? This requires rebuilding the kernel with CONFIG_MODULE_COMPRESS_ZSTD=y. The kmod package which provide depmod is already build with --with-zstd. In can look into this. Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: Hi all, a year ago we changed the default compression and level for the initramfs to zstd -1. This fixed the very slow creation times on development boards (see bug #1958148), but that leads to bigger initramfs sizes that triggered other bugs (like bug #1842320). Big initramfs sizes can also fill up small sized /boot partitions easily (grooming the 850 initramfs-tools bugs revealed several such reports). Using xz -9 would give very good compression, but it takes very long (especially on slow development boards) and a lot of memory (good luck on Raspberry Pis with small memory like Pi Zeros). I propose following approach to address the drawback: Create cpio archives (compressed with xz -9) for the kernel modules and firmware files when building the kernel/firmware Debian package. Then ship those cpio archives in the package (or in a separate binary package). Then the CPU load it put on the builders. The cpio archives would contain the modules for MODULES=most. mkinitramfs will then look for those cpio archives and uses those in case they are present. Such a initramfs would look like this: * AMD/Intel microcode cpio archive (on amd64) * main cpio archive compressed with zstd -1 * kernel modules from the Debian package compressed with xz -9 * firmware files from the Debian package compressed with xz -9 After working on initramfs-tools as part my day job, my fingers were itching and I had to create a quick and dirty draft in my free night time. You can find the result of the last two hours in [1]. This draft has a mkinitramfs-kernel script that creates a cpio archive containing the kernel modules and firmware (that needs to be split later on). The lunar test result on my AMD Ryzen 7 5700G look promising: Building 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but /lib/modules need 54.9 MiB for the cpio archive. The drawback is that building the kernel would take longer, the package takes more space on the archive and mirrors, and downloading them could take longer on slow connections. Implementing my proposal would be relative easy for initramfs-tools, but would mean some work for the kernel team. What do you think? Will the user still be able to add further modules and will machine specific configuration files (e.g. for booting from iSCSI) still be included into the initrd? Best regards Heinrich [1] https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
Hi Heinrich, On Sat, Jul 08, 2023 at 10:44:58AM +0200, Heinrich Schuchardt wrote: > Hello Benjamin, > some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that > a default initrd cannot be decompressed. Some boards may need modules that > are not in initrd by default. Can you clarify where exactly this limitation is? Is this because the system must have room in memory both for the original initramfs image as it is loaded into memory from disk, regardless of compression; plus an uncompressed copy used as the VFS? If that's the case, wouldn't having the kernel modules compressed inside of an uncompressed cpio archive reduce overall memory usage, because the unpacking of the initramfs doesn't uncompress these compressed files? > Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > > > Hi all, > > > > a year ago we changed the default compression and level for the > > initramfs to zstd -1. This fixed the very slow creation times on > > development boards (see bug #1958148), but that leads to bigger > > initramfs sizes that triggered other bugs (like bug #1842320). > > Big initramfs sizes can also fill up small sized /boot partitions easily > > (grooming the 850 initramfs-tools bugs revealed several such reports). > > > > Using xz -9 would give very good compression, but it takes very long > > (especially on slow development boards) and a lot of memory (good luck > > on Raspberry Pis with small memory like Pi Zeros). > > > > I propose following approach to address the drawback: Create cpio > > archives (compressed with xz -9) for the kernel modules and firmware > > files when building the kernel/firmware Debian package. Then ship those > > cpio archives in the package (or in a separate binary package). Then the > > CPU load it put on the builders. The cpio archives would contain the > > modules for MODULES=most. > > > > mkinitramfs will then look for those cpio archives and uses those in > > case they are present. Such a initramfs would look like this: > > > > * AMD/Intel microcode cpio archive (on amd64) > > * main cpio archive compressed with zstd -1 > > * kernel modules from the Debian package compressed with xz -9 > > * firmware files from the Debian package compressed with xz -9 > > > > After working on initramfs-tools as part my day job, my fingers were > > itching and I had to create a quick and dirty draft in my free night > > time. You can find the result of the last two hours in [1]. This draft > > has a mkinitramfs-kernel script that creates a cpio archive containing > > the kernel modules and firmware (that needs to be split later on). > > > > The lunar test result on my AMD Ryzen 7 5700G look promising: Building > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > > /lib/modules need 54.9 MiB for the cpio archive. > > > > The drawback is that building the kernel would take longer, the package > > takes more space on the archive and mirrors, and downloading them could > > take longer on slow connections. > > > > Implementing my proposal would be relative easy for initramfs-tools, but > > would mean some work for the kernel team. > > > > What do you think? > > > > [1] > > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt > > > > -- > > Benjamin Drung > > Debian & Ubuntu Developer -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
Hello Benjamin, some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that a default initrd cannot be decompressed. Some boards may need modules that are not in initrd by default. Best regards Heinrich Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > Hi all, > > a year ago we changed the default compression and level for the > initramfs to zstd -1. This fixed the very slow creation times on > development boards (see bug #1958148), but that leads to bigger > initramfs sizes that triggered other bugs (like bug #1842320). > Big initramfs sizes can also fill up small sized /boot partitions easily > (grooming the 850 initramfs-tools bugs revealed several such reports). > > Using xz -9 would give very good compression, but it takes very long > (especially on slow development boards) and a lot of memory (good luck > on Raspberry Pis with small memory like Pi Zeros). > > I propose following approach to address the drawback: Create cpio > archives (compressed with xz -9) for the kernel modules and firmware > files when building the kernel/firmware Debian package. Then ship those > cpio archives in the package (or in a separate binary package). Then the > CPU load it put on the builders. The cpio archives would contain the > modules for MODULES=most. > > mkinitramfs will then look for those cpio archives and uses those in > case they are present. Such a initramfs would look like this: > > * AMD/Intel microcode cpio archive (on amd64) > * main cpio archive compressed with zstd -1 > * kernel modules from the Debian package compressed with xz -9 > * firmware files from the Debian package compressed with xz -9 > > After working on initramfs-tools as part my day job, my fingers were > itching and I had to create a quick and dirty draft in my free night > time. You can find the result of the last two hours in [1]. This draft > has a mkinitramfs-kernel script that creates a cpio archive containing > the kernel modules and firmware (that needs to be split later on). > > The lunar test result on my AMD Ryzen 7 5700G look promising: Building > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > /lib/modules need 54.9 MiB for the cpio archive. > > The drawback is that building the kernel would take longer, the package > takes more space on the archive and mirrors, and downloading them could > take longer on slow connections. > > Implementing my proposal would be relative easy for initramfs-tools, but > would mean some work for the kernel team. > > What do you think? > > [1] > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt > > -- > Benjamin Drung > Debian & Ubuntu Developer > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 2023-07-08 at 03:23 +0100, Dimitri John Ledkov wrote: > > > On Sat, 8 Jul 2023, 02:49 Benjamin Drung, wrote: > > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung > > > wrote: > > > > > > > > Hi all, > > > > > > > > a year ago we changed the default compression and level for the > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > > development boards (see bug #1958148), but that leads to bigger > > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > > Big initramfs sizes can also fill up small sized /boot > > > > partitions easily > > > > (grooming the 850 initramfs-tools bugs revealed several such > > > > reports). > > > > > > > > Using xz -9 would give very good compression, but it takes very > > > > long > > > > (especially on slow development boards) and a lot of memory > > > > (good luck > > > > on Raspberry Pis with small memory like Pi Zeros). > > > > > > > > I propose following approach to address the drawback: Create > > > > cpio > > > > archives (compressed with xz -9) for the kernel modules and > > > > firmware > > > > files when building the kernel/firmware Debian package. Then > > > > ship those > > > > cpio archives in the package (or in a separate binary package). > > > > Then the > > > > CPU load it put on the builders. The cpio archives would contain > > > > the > > > > modules for MODULES=most. > > > > > > > > mkinitramfs will then look for those cpio archives and uses > > > > those in > > > > case they are present. Such a initramfs would look like this: > > > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > > * main cpio archive compressed with zstd -1 > > > > * kernel modules from the Debian package compressed with xz -9 > > > > * firmware files from the Debian package compressed with xz -9 > > > > > > > > > > Majority of our instances boot without initrd, and there too they > > > don't load most of the modules. > > > Creating xz -9 compressed archive of all modules, still pays the > > > penalty to decompress most of them, and then not modprobe them. > > > I was hoping to achieve a similar in spirit approach, but didn't > > > quite > > > have the time to implement is: > > > > > > 1) change linux-modules and linux-firmware to ship .ko.zst > > > firmware.bin.zst compressed with zstd -19 at .deb build time > > > 2) this saves install size of the packages, with only slightly > > > increased download size > > > 3) modify initramfs-tools to include compressed files into a > > > separate > > > initrd, which is not compressed (i.e. exclude .zst files from the > > > default main compressed cpio archive, and append them in the > > > second > > > main cpio archive that is uncompressed) > > > 4) this should achieve quick initrd creation, which will be > > > smaller in > > > size that current status, and will boot faster as it will only > > > decompress modules/firmware it actually needs at boot > > > > > > For experimentation locally, you can recompress .ko with zstd in > > > place > > > in /lib/modules/; and rerun depmod. To then test initramfs-tools > > > changes that skip over .zst compressed files and add them as is in > > > an > > > uncompressed appended cpio. > > > > That is a very good idea. I created a draft for point 3 in [2]. It > > moves > > the compressed files into a separate directory and creates a > > separate > > cpio archive for that directory without compressing it: > > > > * AMD/Intel microcode cpio archive (on amd64) > > * main cpio archive (compressed) > > * compressed kernel modules / firmware (not compressed) > > > > Sadly this does not work (yet). cpio complains with "premature end > > of > > archive" when looking at it and the kernel fails to extract the last > > cpio part. I am heading to bed now leaving that bug for another day. > > > I had this before with lz4 and lzo compression and fixed this before > in the kernel. It is likely a kernel bug mishandling mixed compression > initrds. It's not only the kernel. It is also cpio that complains. > It will likely work if you Make compressed-files the third early > initrd. Such that we have intel, AMD microcode early initrds, followed > by compressed-files initrd, followed by compressed main initrd. > It did. > > [2] > > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed > > -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote:
> On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote:
> > >
> > > Hi all,
> > >
> > > a year ago we changed the default compression and level for the
> > > initramfs to zstd -1. This fixed the very slow creation times on
> > > development boards (see bug #1958148), but that leads to bigger
> > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > Big initramfs sizes can also fill up small sized /boot partitions easily
> > > (grooming the 850 initramfs-tools bugs revealed several such reports).
> > >
> > > Using xz -9 would give very good compression, but it takes very long
> > > (especially on slow development boards) and a lot of memory (good luck
> > > on Raspberry Pis with small memory like Pi Zeros).
> > >
> > > I propose following approach to address the drawback: Create cpio
> > > archives (compressed with xz -9) for the kernel modules and firmware
> > > files when building the kernel/firmware Debian package. Then ship those
> > > cpio archives in the package (or in a separate binary package). Then the
> > > CPU load it put on the builders. The cpio archives would contain the
> > > modules for MODULES=most.
> > >
> > > mkinitramfs will then look for those cpio archives and uses those in
> > > case they are present. Such a initramfs would look like this:
> > >
> > > * AMD/Intel microcode cpio archive (on amd64)
> > > * main cpio archive compressed with zstd -1
> > > * kernel modules from the Debian package compressed with xz -9
> > > * firmware files from the Debian package compressed with xz -9
> > >
> >
> > Majority of our instances boot without initrd, and there too they
> > don't load most of the modules.
> > Creating xz -9 compressed archive of all modules, still pays the
> > penalty to decompress most of them, and then not modprobe them.
> > I was hoping to achieve a similar in spirit approach, but didn't quite
> > have the time to implement is:
> >
> > 1) change linux-modules and linux-firmware to ship .ko.zst
> > firmware.bin.zst compressed with zstd -19 at .deb build time
> > 2) this saves install size of the packages, with only slightly
> > increased download size
> > 3) modify initramfs-tools to include compressed files into a separate
> > initrd, which is not compressed (i.e. exclude .zst files from the
> > default main compressed cpio archive, and append them in the second
> > main cpio archive that is uncompressed)
> > 4) this should achieve quick initrd creation, which will be smaller in
> > size that current status, and will boot faster as it will only
> > decompress modules/firmware it actually needs at boot
> >
> > For experimentation locally, you can recompress .ko with zstd in place
> > in /lib/modules/; and rerun depmod. To then test initramfs-tools
> > changes that skip over .zst compressed files and add them as is in an
> > uncompressed appended cpio.
>
> That is a very good idea. I created a draft for point 3 in [2]. It moves
> the compressed files into a separate directory and creates a separate
> cpio archive for that directory without compressing it:
>
> * AMD/Intel microcode cpio archive (on amd64)
> * main cpio archive (compressed)
> * compressed kernel modules / firmware (not compressed)
>
> Sadly this does not work (yet). cpio complains with "premature end of
> archive" when looking at it and the kernel fails to extract the last
> cpio part. I am heading to bed now leaving that bug for another day.
>
> [2]
> https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed
Okay. It works now. The not-compressed cpio archive must not be the last
one. So the order is now:
* AMD/Intel microcode cpio archive (on amd64)
* compressed kernel modules / firmware (not compressed)
* main cpio archive (compressed)
I'll really stop now. For a first comparison, the firmware files need to
be converted correctly. There are symlinks in /lib/firmware. So running
following was not correct/enough:
find /lib/firmware -name '*.bin' | while read -r fw; do
sudo zstd -19 -z -o "${fw}.zst" "$fw"
sudo rm "$fw"
done
If you want to help, hand me a correct conversion script.
--
Benjamin Drung
Debian & Ubuntu Developer
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 8 Jul 2023, 02:49 Benjamin Drung, wrote: > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > > > Hi all, > > > > > > a year ago we changed the default compression and level for the > > > initramfs to zstd -1. This fixed the very slow creation times on > > > development boards (see bug #1958148), but that leads to bigger > > > initramfs sizes that triggered other bugs (like bug #1842320). > > > Big initramfs sizes can also fill up small sized /boot partitions > easily > > > (grooming the 850 initramfs-tools bugs revealed several such reports). > > > > > > Using xz -9 would give very good compression, but it takes very long > > > (especially on slow development boards) and a lot of memory (good luck > > > on Raspberry Pis with small memory like Pi Zeros). > > > > > > I propose following approach to address the drawback: Create cpio > > > archives (compressed with xz -9) for the kernel modules and firmware > > > files when building the kernel/firmware Debian package. Then ship those > > > cpio archives in the package (or in a separate binary package). Then > the > > > CPU load it put on the builders. The cpio archives would contain the > > > modules for MODULES=most. > > > > > > mkinitramfs will then look for those cpio archives and uses those in > > > case they are present. Such a initramfs would look like this: > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > * main cpio archive compressed with zstd -1 > > > * kernel modules from the Debian package compressed with xz -9 > > > * firmware files from the Debian package compressed with xz -9 > > > > > > > Majority of our instances boot without initrd, and there too they > > don't load most of the modules. > > Creating xz -9 compressed archive of all modules, still pays the > > penalty to decompress most of them, and then not modprobe them. > > I was hoping to achieve a similar in spirit approach, but didn't quite > > have the time to implement is: > > > > 1) change linux-modules and linux-firmware to ship .ko.zst > > firmware.bin.zst compressed with zstd -19 at .deb build time > > 2) this saves install size of the packages, with only slightly > > increased download size > > 3) modify initramfs-tools to include compressed files into a separate > > initrd, which is not compressed (i.e. exclude .zst files from the > > default main compressed cpio archive, and append them in the second > > main cpio archive that is uncompressed) > > 4) this should achieve quick initrd creation, which will be smaller in > > size that current status, and will boot faster as it will only > > decompress modules/firmware it actually needs at boot > > > > For experimentation locally, you can recompress .ko with zstd in place > > in /lib/modules/; and rerun depmod. To then test initramfs-tools > > changes that skip over .zst compressed files and add them as is in an > > uncompressed appended cpio. > > That is a very good idea. I created a draft for point 3 in [2]. It moves > the compressed files into a separate directory and creates a separate > cpio archive for that directory without compressing it: > > * AMD/Intel microcode cpio archive (on amd64) > * main cpio archive (compressed) > * compressed kernel modules / firmware (not compressed) > > Sadly this does not work (yet). cpio complains with "premature end of > archive" when looking at it and the kernel fails to extract the last > cpio part. I am heading to bed now leaving that bug for another day. > I had this before with lz4 and lzo compression and fixed this before in the kernel. It is likely a kernel bug mishandling mixed compression initrds. It will likely work if you Make compressed-files the third early initrd. Such that we have intel, AMD microcode early initrds, followed by compressed-files initrd, followed by compressed main initrd. > [2] > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed > > -- > Benjamin Drung > Debian & Ubuntu Developer > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > Hi all, > > > > a year ago we changed the default compression and level for the > > initramfs to zstd -1. This fixed the very slow creation times on > > development boards (see bug #1958148), but that leads to bigger > > initramfs sizes that triggered other bugs (like bug #1842320). > > Big initramfs sizes can also fill up small sized /boot partitions easily > > (grooming the 850 initramfs-tools bugs revealed several such reports). > > > > Using xz -9 would give very good compression, but it takes very long > > (especially on slow development boards) and a lot of memory (good luck > > on Raspberry Pis with small memory like Pi Zeros). > > > > I propose following approach to address the drawback: Create cpio > > archives (compressed with xz -9) for the kernel modules and firmware > > files when building the kernel/firmware Debian package. Then ship those > > cpio archives in the package (or in a separate binary package). Then the > > CPU load it put on the builders. The cpio archives would contain the > > modules for MODULES=most. > > > > mkinitramfs will then look for those cpio archives and uses those in > > case they are present. Such a initramfs would look like this: > > > > * AMD/Intel microcode cpio archive (on amd64) > > * main cpio archive compressed with zstd -1 > > * kernel modules from the Debian package compressed with xz -9 > > * firmware files from the Debian package compressed with xz -9 > > > > Majority of our instances boot without initrd, and there too they > don't load most of the modules. > Creating xz -9 compressed archive of all modules, still pays the > penalty to decompress most of them, and then not modprobe them. > I was hoping to achieve a similar in spirit approach, but didn't quite > have the time to implement is: > > 1) change linux-modules and linux-firmware to ship .ko.zst > firmware.bin.zst compressed with zstd -19 at .deb build time > 2) this saves install size of the packages, with only slightly > increased download size > 3) modify initramfs-tools to include compressed files into a separate > initrd, which is not compressed (i.e. exclude .zst files from the > default main compressed cpio archive, and append them in the second > main cpio archive that is uncompressed) > 4) this should achieve quick initrd creation, which will be smaller in > size that current status, and will boot faster as it will only > decompress modules/firmware it actually needs at boot > > For experimentation locally, you can recompress .ko with zstd in place > in /lib/modules/; and rerun depmod. To then test initramfs-tools > changes that skip over .zst compressed files and add them as is in an > uncompressed appended cpio. That is a very good idea. I created a draft for point 3 in [2]. It moves the compressed files into a separate directory and creates a separate cpio archive for that directory without compressing it: * AMD/Intel microcode cpio archive (on amd64) * main cpio archive (compressed) * compressed kernel modules / firmware (not compressed) Sadly this does not work (yet). cpio complains with "premature end of archive" when looking at it and the kernel fails to extract the last cpio part. I am heading to bed now leaving that bug for another day. [2] https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Reducing initramfs size and speed up the generation
On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > Hi all, > > a year ago we changed the default compression and level for the > initramfs to zstd -1. This fixed the very slow creation times on > development boards (see bug #1958148), but that leads to bigger > initramfs sizes that triggered other bugs (like bug #1842320). > Big initramfs sizes can also fill up small sized /boot partitions easily > (grooming the 850 initramfs-tools bugs revealed several such reports). > > Using xz -9 would give very good compression, but it takes very long > (especially on slow development boards) and a lot of memory (good luck > on Raspberry Pis with small memory like Pi Zeros). > > I propose following approach to address the drawback: Create cpio > archives (compressed with xz -9) for the kernel modules and firmware > files when building the kernel/firmware Debian package. Then ship those > cpio archives in the package (or in a separate binary package). Then the > CPU load it put on the builders. The cpio archives would contain the > modules for MODULES=most. > > mkinitramfs will then look for those cpio archives and uses those in > case they are present. Such a initramfs would look like this: > > * AMD/Intel microcode cpio archive (on amd64) > * main cpio archive compressed with zstd -1 > * kernel modules from the Debian package compressed with xz -9 > * firmware files from the Debian package compressed with xz -9 > Majority of our instances boot without initrd, and there too they don't load most of the modules. Creating xz -9 compressed archive of all modules, still pays the penalty to decompress most of them, and then not modprobe them. I was hoping to achieve a similar in spirit approach, but didn't quite have the time to implement is: 1) change linux-modules and linux-firmware to ship .ko.zst firmware.bin.zst compressed with zstd -19 at .deb build time 2) this saves install size of the packages, with only slightly increased download size 3) modify initramfs-tools to include compressed files into a separate initrd, which is not compressed (i.e. exclude .zst files from the default main compressed cpio archive, and append them in the second main cpio archive that is uncompressed) 4) this should achieve quick initrd creation, which will be smaller in size that current status, and will boot faster as it will only decompress modules/firmware it actually needs at boot For experimentation locally, you can recompress .ko with zstd in place in /lib/modules/; and rerun depmod. To then test initramfs-tools changes that skip over .zst compressed files and add them as is in an uncompressed appended cpio. > After working on initramfs-tools as part my day job, my fingers were > itching and I had to create a quick and dirty draft in my free night > time. You can find the result of the last two hours in [1]. This draft > has a mkinitramfs-kernel script that creates a cpio archive containing > the kernel modules and firmware (that needs to be split later on). > > The lunar test result on my AMD Ryzen 7 5700G look promising: Building > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is > 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but > /lib/modules need 54.9 MiB for the cpio archive. > > The drawback is that building the kernel would take longer, the package > takes more space on the archive and mirrors, and downloading them could > take longer on slow connections. > > Implementing my proposal would be relative easy for initramfs-tools, but > would mean some work for the kernel team. > > What do you think? > > [1] > https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt > > -- > Benjamin Drung > Debian & Ubuntu Developer > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- okurrr, Dimitri -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reducing initramfs size and speed up the generation
Hi all, a year ago we changed the default compression and level for the initramfs to zstd -1. This fixed the very slow creation times on development boards (see bug #1958148), but that leads to bigger initramfs sizes that triggered other bugs (like bug #1842320). Big initramfs sizes can also fill up small sized /boot partitions easily (grooming the 850 initramfs-tools bugs revealed several such reports). Using xz -9 would give very good compression, but it takes very long (especially on slow development boards) and a lot of memory (good luck on Raspberry Pis with small memory like Pi Zeros). I propose following approach to address the drawback: Create cpio archives (compressed with xz -9) for the kernel modules and firmware files when building the kernel/firmware Debian package. Then ship those cpio archives in the package (or in a separate binary package). Then the CPU load it put on the builders. The cpio archives would contain the modules for MODULES=most. mkinitramfs will then look for those cpio archives and uses those in case they are present. Such a initramfs would look like this: * AMD/Intel microcode cpio archive (on amd64) * main cpio archive compressed with zstd -1 * kernel modules from the Debian package compressed with xz -9 * firmware files from the Debian package compressed with xz -9 After working on initramfs-tools as part my day job, my fingers were itching and I had to create a quick and dirty draft in my free night time. You can find the result of the last two hours in [1]. This draft has a mkinitramfs-kernel script that creates a cpio archive containing the kernel modules and firmware (that needs to be split later on). The lunar test result on my AMD Ryzen 7 5700G look promising: Building 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but /lib/modules need 54.9 MiB for the cpio archive. The drawback is that building the kernel would take longer, the package takes more space on the archive and mirrors, and downloading them could take longer on slow connections. Implementing my proposal would be relative easy for initramfs-tools, but would mean some work for the kernel team. What do you think? [1] https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
