Re: Amendment: Update on reducing initramfs size and speed up
On Thu, 27 Jul 2023 at 00:51, Michael Hudson-Doyle wrote: > > > > On Thu, 27 Jul 2023 at 09:21, Benjamin Drung wrote: >> >> On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote: >> > Hi all, >> > >> > A few weeks ago, I posted an idea how to reduce the initramfs size and >> > speed up the generation: >> > >> > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html >> > >> > This post sparked a lively discussion. The initial idea was ditched for >> > a better solution: mkinitramfs will put all compressed files (kernel >> > modules and firmware files) into a cpio archive that is not compressed >> > (because compressing compressed files makes no sense). All other files >> > will be added to a cpio archive that gets compressed. As next steps, the >> > kernel modules and firmware files need to be shipped compressed. >> > >> > After several iterations for the implementation and review by Daves >> > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that puts >> > compressed kernel modules and firmware files in an uncompressed cpio >> > (https://launchpad.net/bugs/2028567). >> > >> > I created/updated the follow-up tickets and added my patches to them: >> > >> > Ship kernel modules Zstd compressed >> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568 >> > >> > compress firmware in /lib/firmware >> > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260 >> > >> > And without further ado, here come the benchmark results: >> > >> > The benchmarks were done either on an AMD Ryzen 7 5700G with schroot and >> > overlay on tmpfs or on the hardware mentioned. All tests were running >> > the latest Ubuntu mantic development release. >> > >> > * minimal: schroot with linux-image-generic initramfs-tools zstd >> > * full: minimal + busybox-initramfs cryptsetup-initramfs >> > isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-spinner >> > * nvidia: full + linux-headers-generic nvidia-driver-525 >> > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/ >> > * VisionFive 2: VisionFive 2 RISC-V board >> > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf) >> > >> > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa i.e. >> > * initramfs-tools 0.142ubuntu7bd4 >> > * linux 6.3.0-7.7bd2 >> > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1 >> > >> > > || build | size | uncompressed size | >> > > | test | time| in bytes | in MiB | in bytes | in MiB | >> > > ||-|---||| >> > > | minimal| 4.30 s | 66701818 | 63.6 | 224087608 | 213.7 | >> > > | minimal next | 4.54 s | 59935186 | 57.2 | 67738810 | 64.6 | >> > > | full | 7.15 s | 118007038 | 112.5 | 387976843 | 370.0 | >> > > | full next | 7.29 s | 106937908 | 102.0 | 128610985 | 122.7 | >> > > | nvidia | 7.04 s | 209200523 | 199.5 | 513554279 | 489.8 | >> > > | nvidia next| 7.21 s | 195246287 | 186.2 | 235288174 | 224.4 | >> > > | nvidia fw next | 7.16 s | 191329102 | 182.5 | 213078234 | 203.2 | >> > > | VisionFive 2 | 142.9 s | 121895035 | 116.2 | 411160836 | 392.1 | >> > > | VF 2 next | 126.7 s | 111651453 | 106.5 | 134120804 | 127.9 | >> > > | RPi Zero 2 | 109.5 s | 39803044 | 40.0 | 69592789 | 66.4 | >> > > | RPi Zero 2 ² | 103.5 s | 39804499 | 40.0 | 69592789 | 66.4 | >> > > | RPi Zero 2 next| 101.2 s | 31463352 | 30.0 | 41145762 | 39.2 | >> > >> > ² Updated initramfs-tools (but no compressed modules or firmware) >> > >> > The build time was averaged over five runs. >> > >> > > | improvement | build time | size | uncompressed size | >> > > |--|||---| >> > > | minimal | 105.6 % | 89.9 % | 30.2 % | >> > > | full | 102.0 % | 90.6 % | 33.1 % | >> > > | nvidia | 101.7 % | 91.5 % | 41.5 % | >> > > | VisionFive 2 | 88.7 % | 91.6 % | 32.6 % | >> > > | RPi Zero 2 | 92.4 % | 79.0 % | 59.1 % | >> > >> > Building the initramfs takes more CPU cycles (see tests on tmpfs), but >> > saves time on disk IO. Daves Jones saw much bigger time savings on his >> > Raspberry Pis but his tests were on lunar. >> > >> > Build time influence: >> > + add_directories plus uniq take several milliseconds >> > + depmod on compressed kernel modules take hundreds of >> > milliseconds longer >> > - copying smaller kernel modules (due to compression) is faster >> > - cpio archive that needs to be compressed is smaller >> > - not storing intermediate cpio archives saves time >> > >> > Saving 10 to 20 percent on the initramfs size and only needing half or a >> > third of the size when unpacked (i.e. needed memory during boot) is a >> > good improvement. >> >> The smaller initramfs overall size (less to load into memory and unpack) >> and the smaller compressed cpio
desktop-file-utils (0.26-1ubuntu5) desktop spec version 1.5
Hello, It appears the desktop-file-utils package does not support desktop file spec version 1.5: $ desktop-file-validate Applications/squashfs-root/org.keepassxc.KeePassXC.desktop Applications/squashfs-root/org.keepassxc.KeePassXC.desktop: error: value "1.5" for key "Version" in group "Desktop Entry" is not a known version Applications/squashfs-root/org.keepassxc.KeePassXC.desktop: hint: value item "Security" in key "Categories" in group "Desktop Entry" can be extended with another category among the following categories: Settings, or System Applications/squashfs-root/org.keepassxc.KeePassXC.desktop: error: file contains key "SingleMainWindow" in group "Desktop Entry", but keys extending the format should start with "X-" This was fixed upstream in commit https://gitlab.freedesktop.org/xdg/desktop-file-utils/-/commit/56d220dd679c7c3a8f995a41a27a7d6f3df49dea in 2022, can these changes be applied? I also filed this here https://bugs.launchpad.net/ubuntu/+source/desktop-file-utils/+bug/2028312 Regards, Alex-- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Amendment: Update on reducing initramfs size and speed up
On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote: > Hi all, > > A few weeks ago, I posted an idea how to reduce the initramfs size and > speed up the generation: > > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html > > This post sparked a lively discussion. The initial idea was ditched for > a better solution: mkinitramfs will put all compressed files (kernel > modules and firmware files) into a cpio archive that is not compressed > (because compressing compressed files makes no sense). All other files > will be added to a cpio archive that gets compressed. As next steps, the > kernel modules and firmware files need to be shipped compressed. > > After several iterations for the implementation and review by Daves > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that puts > compressed kernel modules and firmware files in an uncompressed cpio > (https://launchpad.net/bugs/2028567). > > I created/updated the follow-up tickets and added my patches to them: > > Ship kernel modules Zstd compressed > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568 > > compress firmware in /lib/firmware > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260 > > And without further ado, here come the benchmark results: > > The benchmarks were done either on an AMD Ryzen 7 5700G with schroot and > overlay on tmpfs or on the hardware mentioned. All tests were running > the latest Ubuntu mantic development release. > > * minimal: schroot with linux-image-generic initramfs-tools zstd > * full: minimal + busybox-initramfs cryptsetup-initramfs > isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-spinner > * nvidia: full + linux-headers-generic nvidia-driver-525 > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/ > * VisionFive 2: VisionFive 2 RISC-V board > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf) > > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa i.e. > * initramfs-tools 0.142ubuntu7bd4 > * linux 6.3.0-7.7bd2 > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1 > > > || build | size | uncompressed size | > > | test | time| in bytes | in MiB | in bytes | in MiB | > > ||-|---||| > > | minimal| 4.30 s | 66701818 | 63.6 | 224087608 | 213.7 | > > | minimal next | 4.54 s | 59935186 | 57.2 | 67738810 | 64.6 | > > | full | 7.15 s | 118007038 | 112.5 | 387976843 | 370.0 | > > | full next | 7.29 s | 106937908 | 102.0 | 128610985 | 122.7 | > > | nvidia | 7.04 s | 209200523 | 199.5 | 513554279 | 489.8 | > > | nvidia next| 7.21 s | 195246287 | 186.2 | 235288174 | 224.4 | > > | nvidia fw next | 7.16 s | 191329102 | 182.5 | 213078234 | 203.2 | > > | VisionFive 2 | 142.9 s | 121895035 | 116.2 | 411160836 | 392.1 | > > | VF 2 next | 126.7 s | 111651453 | 106.5 | 134120804 | 127.9 | > > | RPi Zero 2 | 109.5 s | 39803044 | 40.0 | 69592789 | 66.4 | > > | RPi Zero 2 ² | 103.5 s | 39804499 | 40.0 | 69592789 | 66.4 | > > | RPi Zero 2 next| 101.2 s | 31463352 | 30.0 | 41145762 | 39.2 | > > ² Updated initramfs-tools (but no compressed modules or firmware) > > The build time was averaged over five runs. > > > | improvement | build time | size | uncompressed size | > > |--|||---| > > | minimal | 105.6 % | 89.9 % | 30.2 % | > > | full | 102.0 % | 90.6 % | 33.1 % | > > | nvidia | 101.7 % | 91.5 % | 41.5 % | > > | VisionFive 2 | 88.7 % | 91.6 % | 32.6 % | > > | RPi Zero 2 | 92.4 % | 79.0 % | 59.1 % | > > Building the initramfs takes more CPU cycles (see tests on tmpfs), but > saves time on disk IO. Daves Jones saw much bigger time savings on his > Raspberry Pis but his tests were on lunar. > > Build time influence: > + add_directories plus uniq take several milliseconds > + depmod on compressed kernel modules take hundreds of > milliseconds longer > - copying smaller kernel modules (due to compression) is faster > - cpio archive that needs to be compressed is smaller > - not storing intermediate cpio archives saves time > > Saving 10 to 20 percent on the initramfs size and only needing half or a > third of the size when unpacked (i.e. needed memory during boot) is a > good improvement. The smaller initramfs overall size (less to load into memory and unpack) and the smaller compressed cpio (less to decompress) have a positive effect on the boot speed, especially on systems with slow CPU and/or slow IO. When looking at the "kernel" time from systemd-analyze, the improvement ranges from 1.62s - 1.36s = 0.26s in a VM on my desktop to a heavily noticeable 37.9s - 16.5s = 21.4s on the VisionFive 2 RISC-V board. -- Benjamin Drung Debian & Ubuntu Developer --
Update on reducing initramfs size and speed up
Hi all, A few weeks ago, I posted an idea how to reduce the initramfs size and speed up the generation: https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html This post sparked a lively discussion. The initial idea was ditched for a better solution: mkinitramfs will put all compressed files (kernel modules and firmware files) into a cpio archive that is not compressed (because compressing compressed files makes no sense). All other files will be added to a cpio archive that gets compressed. As next steps, the kernel modules and firmware files need to be shipped compressed. After several iterations for the implementation and review by Daves Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that puts compressed kernel modules and firmware files in an uncompressed cpio (https://launchpad.net/bugs/2028567). I created/updated the follow-up tickets and added my patches to them: Ship kernel modules Zstd compressed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568 compress firmware in /lib/firmware https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260 And without further ado, here come the benchmark results: The benchmarks were done either on an AMD Ryzen 7 5700G with schroot and overlay on tmpfs or on the hardware mentioned. All tests were running the latest Ubuntu mantic development release. * minimal: schroot with linux-image-generic initramfs-tools zstd * full: minimal + busybox-initramfs cryptsetup-initramfs isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-spinner * nvidia: full + linux-headers-generic nvidia-driver-525 * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/ * VisionFive 2: VisionFive 2 RISC-V board * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf) "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa i.e. * initramfs-tools 0.142ubuntu7bd4 * linux 6.3.0-7.7bd2 * linux-firmware 20230629.gitee91452d-0ubuntu1bd1 || build | size | uncompressed size | | test | time| in bytes | in MiB | in bytes | in MiB | ||-|---||| | minimal| 4.30 s | 66701818 | 63.6 | 224087608 | 213.7 | | minimal next | 4.54 s | 59935186 | 57.2 | 67738810 | 64.6 | | full | 7.15 s | 118007038 | 112.5 | 387976843 | 370.0 | | full next | 7.29 s | 106937908 | 102.0 | 128610985 | 122.7 | | nvidia | 7.04 s | 209200523 | 199.5 | 513554279 | 489.8 | | nvidia next| 7.21 s | 195246287 | 186.2 | 235288174 | 224.4 | | nvidia fw next | 7.16 s | 191329102 | 182.5 | 213078234 | 203.2 | | VisionFive 2 | 142.9 s | 121895035 | 116.2 | 411160836 | 392.1 | | VF 2 next | 126.7 s | 111651453 | 106.5 | 134120804 | 127.9 | | RPi Zero 2 | 109.5 s | 39803044 | 40.0 | 69592789 | 66.4 | | RPi Zero 2 ² | 103.5 s | 39804499 | 40.0 | 69592789 | 66.4 | | RPi Zero 2 next| 101.2 s | 31463352 | 30.0 | 41145762 | 39.2 | ² Updated initramfs-tools (but no compressed modules or firmware) The build time was averaged over five runs. | improvement | build time | size | uncompressed size | |--|||---| | minimal | 105.6 % | 89.9 % | 30.2 % | | full | 102.0 % | 90.6 % | 33.1 % | | nvidia | 101.7 % | 91.5 % | 41.5 % | | VisionFive 2 | 88.7 % | 91.6 % | 32.6 % | | RPi Zero 2 | 92.4 % | 79.0 % | 59.1 % | Building the initramfs takes more CPU cycles (see tests on tmpfs), but saves time on disk IO. Daves Jones saw much bigger time savings on his Raspberry Pis but his tests were on lunar. Build time influence: + add_directories plus uniq take several milliseconds + depmod on compressed kernel modules take hundreds of milliseconds longer - copying smaller kernel modules (due to compression) is faster - cpio archive that needs to be compressed is smaller - not storing intermediate cpio archives saves time Saving 10 to 20 percent on the initramfs size and only needing half or a third of the size when unpacked (i.e. needed memory during boot) is a good improvement. -- Benjamin Drung Debian & Ubuntu Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
