Re: Reducing initramfs size and speed up the generation

2023-08-11 Thread Steve Langasek
On Wed, Aug 09, 2023 at 05:30:04PM +, Benjamin Drung wrote: > > The benefits to this are that the firmware and base initrds only need be > > generated once regardless of number of kernels installed. And their > > generation is decoupled from kernel upgrades/installs and each other. > > So the

Re: Reducing initramfs size and speed up the generation

2023-08-09 Thread Benjamin Drung
Hi, On Wed, 2023-08-09 at 01:22 -0500, Glenn Washburn wrote: > Hi, > > I recently came across the thread last month of the same name as the > subject and would like to bring up for discussion some ideas orthagonal > to all the ones presented there. > > Some observations: > 1. I have observed

Re: Reducing initramfs size and speed up the generation

2023-07-25 Thread Dimitri John Ledkov
On Sat, 22 Jul 2023 at 02:03, Steve Langasek wrote: > > On Tue, Jul 11, 2023 at 01:17:27AM +0200, Benjamin Drung wrote: > > If you want to test it yourself, you can find initramfs-tools > > 0.142ubuntu7bd2 for mantic in my PPA: > > https://launchpad.net/~bdrung/+archive/ubuntu/ppa > > What blocks

Re: Reducing initramfs size and speed up the generation

2023-07-25 Thread Dimitri John Ledkov
On Fri, 21 Jul 2023 at 21:08, Steve Langasek wrote: > > On Fri, Jul 21, 2023 at 07:33:58PM +0100, Dimitri John Ledkov wrote: > > If it is a concern that v5.15 jammy kernel may potentially be used > > after partial / incomplete upgrade to Mantic, we can opt into using XZ > > compressed firmware or

Re: Reducing initramfs size and speed up the generation

2023-07-24 Thread Dimitri John Ledkov
On Mon, 24 Jul 2023 at 13:34, Adrien Nader wrote: > > On Tue, Jul 11, 2023, Michael Hudson-Doyle wrote: > > I was wondering if it make sense to construct a zstd dictionary for > > compressing kernel modules but I didn't realize they need to be available > > at decompression time, I'm not sure the

Re: Reducing initramfs size and speed up the generation

2023-07-21 Thread Steve Langasek
On Tue, Jul 11, 2023 at 01:17:27AM +0200, Benjamin Drung wrote: > If you want to test it yourself, you can find initramfs-tools > 0.142ubuntu7bd2 for mantic in my PPA: > https://launchpad.net/~bdrung/+archive/ubuntu/ppa What blocks this from being uploaded to mantic? AFAICS the behavior should

Re: Reducing initramfs size and speed up the generation

2023-07-21 Thread Steve Langasek
On Fri, Jul 21, 2023 at 07:33:58PM +0100, Dimitri John Ledkov wrote: > If it is a concern that v5.15 jammy kernel may potentially be used > after partial / incomplete upgrade to Mantic, we can opt into using XZ > compressed firmware or I can backport ZSTD firmware support to jammy > GA kernel - it

Re: Reducing initramfs size and speed up the generation

2023-07-21 Thread Dimitri John Ledkov
On Fri, 21 Jul 2023 at 17:59, Steve Langasek wrote: > > On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote: > > > > Okay. It works now. The not-compressed cpio archive must not be the last > > > > one. So the order is now: > > > > > > > > * AMD/Intel microcode cpio archive (on amd64)

Re: Reducing initramfs size and speed up the generation

2023-07-21 Thread Steve Langasek
On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote: > > > Okay. It works now. The not-compressed cpio archive must not be the last > > > one. So the order is now: > > > > > > * AMD/Intel microcode cpio archive (on amd64) > > > * compressed kernel modules / firmware (not compressed) >

Re: Reducing initramfs size and speed up the generation

2023-07-14 Thread TJ
On 13/07/2023 22:36, Benjamin Drung wrote: One small change that brings major benefits is only including the firmware files required, not every file declared by a kernel module. The GPU drivers are the main culprit there with amdgpu declaring over 500 of which, for any individual GPU, only a

Re: Reducing initramfs size and speed up the generation

2023-07-14 Thread TJ
On 13/07/2023 23:30, Dimitri John Ledkov wrote: journald stores a copy of dmesg subject to its own retention and rotation policy, even when dmesg is cleared. Thus it should be reliable to have journal output for dmesg for multiple boots. The issue however is that drivers and kernels can change

Re: Reducing initramfs size and speed up the generation

2023-07-13 Thread Dimitri John Ledkov
On Thu, 13 Jul 2023, 22:37 Benjamin Drung, wrote: > On Thu, 2023-07-13 at 08:32 +0100, TJ wrote: > > Is there a specific reason why the focus is on trying to shoe-horn > > everything possible into the initrd.img and then compress rather than > > winnow out the files an installed system will

Re: Reducing initramfs size and speed up the generation

2023-07-13 Thread Benjamin Drung
On Thu, 2023-07-13 at 08:32 +0100, TJ wrote: > Is there a specific reason why the focus is on trying to shoe-horn > everything possible into the initrd.img and then compress rather than > winnow out the files an installed system will never need to find the > root file-system? These are two

Re: Reducing initramfs size and speed up the generation

2023-07-13 Thread Adrien Nader
On Thu, Jul 13, 2023, Julian Andres Klode wrote: > On Thu, Jul 13, 2023 at 08:32:48AM +0100, TJ wrote: > > Is there a specific reason why the focus is on trying to shoe-horn > > everything possible into the initrd.img and then compress rather than winnow > > out the files an installed system will

Re: Reducing initramfs size and speed up the generation

2023-07-13 Thread Julian Andres Klode
On Thu, Jul 13, 2023 at 08:32:48AM +0100, TJ wrote: > Is there a specific reason why the focus is on trying to shoe-horn > everything possible into the initrd.img and then compress rather than winnow > out the files an installed system will never need to find the root > file-system? Basically the

Re: Reducing initramfs size and speed up the generation

2023-07-13 Thread TJ
Is there a specific reason why the focus is on trying to shoe-horn everything possible into the initrd.img and then compress rather than winnow out the files an installed system will never need to find the root file-system? I tackled the issue of ever-expanding host-generated initrd.img file

Re: Reducing initramfs size and speed up the generation

2023-07-11 Thread Adrien Nader
On Tue, Jul 11, 2023, Seth Arnold wrote: > On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > > There is a little-know but very interesting property of LZMA: its > > decompression speed does not depend on the uncompressed size but only on > > the compressed size. What this means is

Re: Reducing initramfs size and speed up the generation

2023-07-11 Thread Julian Andres Klode
On Tue, Jul 11, 2023 at 12:53:00AM +, Seth Arnold wrote: > On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > > There is a little-know but very interesting property of LZMA: its > > decompression speed does not depend on the uncompressed size but only on > > the compressed size.

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Michael Hudson-Doyle
On Tue, 11 Jul 2023 at 11:17, Benjamin Drung wrote: > On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote: > > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, > 02:19: > > > > > > > > Hi all, > > > > > > > > a year

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Seth Arnold
On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote: > There is a little-know but very interesting property of LZMA: its > decompression speed does not depend on the uncompressed size but only on > the compressed size. What this means is that if you compress a 100MB > down to 20MB, it

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Benjamin Drung
On Mon, 2023-07-10 at 11:09 +0200, Adrien Nader wrote: > On Sat, Jul 08, 2023, Benjamin Drung wrote: > > On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote: > > > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > >

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Benjamin Drung
On Mon, 2023-07-10 at 16:12 +0200, Heinrich Schuchardt wrote: > On 10.07.23 00:29, Steve Langasek wrote: > > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, > > > > > 02:19: > > > > > > > > Hi all, > > > > > > > > a

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Benjamin Drung
On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote: > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > > > > > > Hi all, > > > > > > a year ago we changed the default compression and level for the > > > > >

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Adrien Nader
On Sat, Jul 08, 2023, Seth Arnold wrote: > On Sat, Jul 08, 2023 at 02:18:52AM +0200, Benjamin Drung wrote: > > Using xz -9 would give very good compression, but it takes very long > > (especially on slow development boards) and a lot of memory (good luck > > on Raspberry Pis with small memory like

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Adrien Nader
On Sat, Jul 08, 2023, Benjamin Drung wrote: > On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote: > > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > > > > > Hi all, > > > > > > > > a year ago we changed the

Re: Reducing initramfs size and speed up the generation

2023-07-10 Thread Heinrich Schuchardt
On 10.07.23 00:29, Steve Langasek wrote: On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: Hi all, a year ago we changed the default compression and level for the initramfs to zstd -1. This fixed the very slow

Re: Reducing initramfs size and speed up the generation

2023-07-09 Thread Steve Langasek
On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote: > > > Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > > > > Hi all, > > > > a year ago we changed the default compression and level for the > > > > initramfs to zstd -1. This fixed the very slow creation times on > > > >

Re: Reducing initramfs size and speed up the generation

2023-07-08 Thread Heinrich Schuchardt
On 7/8/23 21:42, Steve Langasek wrote: Hi Heinrich, On Sat, Jul 08, 2023 at 10:44:58AM +0200, Heinrich Schuchardt wrote: Hello Benjamin, some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that a default initrd cannot be decompressed. Some boards may need modules that

Re: Reducing initramfs size and speed up the generation

2023-07-08 Thread Steve Langasek
Hi Heinrich, On Sat, Jul 08, 2023 at 10:44:58AM +0200, Heinrich Schuchardt wrote: > Hello Benjamin, > some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that > a default initrd cannot be decompressed. Some boards may need modules that > are not in initrd by default. Can you

Re: Reducing initramfs size and speed up the generation

2023-07-08 Thread Heinrich Schuchardt
Hello Benjamin, some of the Ubuntu supported boards (e.g. LicheeRV) have so little RAM that a default initrd cannot be decompressed. Some boards may need modules that are not in initrd by default. Best regards Heinrich Benjamin Drung schrieb am Sa., 8. Juli 2023, 02:19: > Hi all, > > a year

Re: Reducing initramfs size and speed up the generation

2023-07-07 Thread Benjamin Drung
On Sat, 2023-07-08 at 03:23 +0100, Dimitri John Ledkov wrote: > > > On Sat, 8 Jul 2023, 02:49 Benjamin Drung, wrote: > > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung > > > wrote: > > > > > > > > Hi all, > > > > > > > > a year

Re: Reducing initramfs size and speed up the generation

2023-07-07 Thread Benjamin Drung
On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote: > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > > > Hi all, > > > > > > a year ago we changed the default compression and level for the > > > initramfs to zstd

Re: Reducing initramfs size and speed up the generation

2023-07-07 Thread Dimitri John Ledkov
On Sat, 8 Jul 2023, 02:49 Benjamin Drung, wrote: > On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > > > Hi all, > > > > > > a year ago we changed the default compression and level for the > > > initramfs to zstd -1.

Re: Reducing initramfs size and speed up the generation

2023-07-07 Thread Benjamin Drung
On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote: > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > > > Hi all, > > > > a year ago we changed the default compression and level for the > > initramfs to zstd -1. This fixed the very slow creation times on > > development boards

Re: Reducing initramfs size and speed up the generation

2023-07-07 Thread Dimitri John Ledkov
On Sat, 8 Jul 2023 at 01:19, Benjamin Drung wrote: > > Hi all, > > a year ago we changed the default compression and level for the > initramfs to zstd -1. This fixed the very slow creation times on > development boards (see bug #1958148), but that leads to bigger > initramfs sizes that triggered