Re: Kernel modules in initrd

[Ludovic Courtès]

Danny Milosavljevic  skribis:

> On Tue, 27 Feb 2018 20:32:49 +0100
> Danny Milosavljevic  wrote:
>
>> > > P.S. How come glibc is in the initrd?  Shouldn't guile have statically 
>> > > linked it?
>> > > glibc is like 5 kiB.  In that case saving 800 kiB is not really 
>> > > worth it...
>> > 
>> > One of the packages that ends up in the initrd must be dynamically
>> > linked.  You need to find out which one it is.  
>> 
>> It's because of bash-minimal.
>
> And that's because of:
>
> ./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:
>   (open-pipe* mode 
> "/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" 
> "-c" command))

Ouch, indeed!

--8<---cut here---start->8---
$ guix size guile-static-stripped
store item   totalself
/gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.377.0 
   45.7  59.4%
/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c30.3  
  28.8  37.4%
/gnu/store/05dvazr5wfh7lxx4zi54zfqnx6ha8vxr-bash-static-4.4.12   1.5 
1.5   1.9%
/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12 31.3 
1.0   1.3%
total: 77.0 MiB
--8<---cut here---end--->8---

How long have we had this?

I’ve fixed it in 63e48300d19b848dfe75880581338c2c73b7b0df, so we’re now
at 45.7 MiB, which is still a lot, but much better.

Ludo'.

Re: Kernel modules in initrd

[Danny Milosavljevic]

On Tue, 27 Feb 2018 20:32:49 +0100
Danny Milosavljevic  wrote:

> > > P.S. How come glibc is in the initrd?  Shouldn't guile have statically 
> > > linked it?
> > > glibc is like 5 kiB.  In that case saving 800 kiB is not really worth 
> > > it...
> > 
> > One of the packages that ends up in the initrd must be dynamically
> > linked.  You need to find out which one it is.  
> 
> It's because of bash-minimal.

And that's because of:

./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:
  (open-pipe* mode 
"/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" 
command))

Re: Kernel modules in initrd

[Danny Milosavljevic]

> > P.S. How come glibc is in the initrd?  Shouldn't guile have statically 
> > linked it?
> > glibc is like 5 kiB.  In that case saving 800 kiB is not really worth 
> > it...  
> 
> One of the packages that ends up in the initrd must be dynamically
> linked.  You need to find out which one it is.

It's because of bash-minimal.

$ grep -r 'bash-minimal' .
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guild:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guile-config:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guile-snarf:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
Binary file 
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/lib/guile/2.2/ccache/ice-9/popen.go
 matches
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/share/guile/2.2/ice-9/popen.scm:
  (open-pipe* mode 
"/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" 
command))
Binary file 
./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash 
matches
./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bashbug:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
 -
./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bashbug:CFLAGS="
 -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' 
-DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -DCONF_VENDOR='unknown' 
-DLOCALEDIR='/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/share/locale'
 -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -g 
-O2 -Wno-parentheses -Wno-format-security"
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzdiff:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzgrep:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzless:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzmore:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
Binary file 
./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/lib/guile/2.2/ccache/ice-9/popen.go
 matches
./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:
  (open-pipe* mode 
"/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" 
command))

Re: Kernel modules in initrd

[Ludovic Courtès]

Hello,

ng0  skribis:

> Ludovic Courtès transcribed 0.8K bytes:
>> Hello,
>> 
>> Mark H Weaver  skribis:
>> 
>> > To make this easier, I think the right approach is to include many
>> > modules like these to our installation image initrd, and then to
>> > automatically detect which modules are needed for booting.  A future
>> > easy installer could automatically add those modules to the OS config,
>> > but in the meantime we could simply print a message recommending that
>> > the user should add the needed modules to their initrd config.
>> 
>> FWIW I have code that I was planning to polish and submit (hopefully
>> next week) that determines modules needed for a particular device by
>> walking /sys.  In addition, I thought we could make the initrd module
>> list a first class field of ‘operating-system’.  With these two pieces,
>> ‘guix system’ should be able to warn when a module is missing.
>> 
>> Ludo’.
>
> I'm willing to test those, it's probably less time consuming than figuring out
> the corretc modules the Hetzner CLoud instance needs.

I’ve posted the patch series here:

  https://bugs.gnu.org/30629

Feedback welcome!

Ludo’.

Re: Kernel modules in initrd

[Ludovic Courtès]

Danny Milosavljevic  skribis:

> So what we would need next is something like modprobe written in guile.
>
> I think that ./gnu/build/linux-modules.scm load-linux-module* already does 
> that.
>
> The only part missing is to replace %modprobe-wrapper by a guile script
> which imports ./gnu/build/linux-modules.scm load-linux-module*, interprets
> some of the modprobe options and then masquerades as modprobe without invoking
> modprobe.
>
> The reason is that the kernel calls us back when it needs a module (for
> example when you mount something and it needs a charset converter or 
> something).
>
> (the name of the executable invoked is configured via 
> /proc/sys/kernel/modprobe)
>
> This can happen quite early in the boot process - and it took me some time
> to get the order and also the environment variables right.  We should be able
> to reuse that for the pure Guile modprobe.
>
> After the patch we don't *manually* load any of the modules - and all the
> tests work.  I booted my system with it - that also works.

I see, that’s a workable plan, we should do that!

> P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked 
> it?
> glibc is like 5 kiB.  In that case saving 800 kiB is not really worth 
> it...

One of the packages that ends up in the initrd must be dynamically
linked.  You need to find out which one it is.

Thanks for the explanations!

Ludo’.

Re: Kernel modules in initrd

[Danny Milosavljevic]

Hi Ludo,

On Mon, 26 Feb 2018 16:20:08 +0100
l...@gnu.org (Ludovic Courtès) wrote:

> I suppose the bits I’ve promised to detect necessary modules based on
> modules.alias and /sys should be useful here?

Definitely!

I've posted a working patch series (v3, "Load Linux module only when supported
hardware is present") of the current kmod-based autoloader to guix-patches.
It's using an old-ish kmod - but otherwise it works fine.

So what we would need next is something like modprobe written in guile.

I think that ./gnu/build/linux-modules.scm load-linux-module* already does that.

The only part missing is to replace %modprobe-wrapper by a guile script
which imports ./gnu/build/linux-modules.scm load-linux-module*, interprets
some of the modprobe options and then masquerades as modprobe without invoking
modprobe.

The reason is that the kernel calls us back when it needs a module (for
example when you mount something and it needs a charset converter or something).

(the name of the executable invoked is configured via /proc/sys/kernel/modprobe)

This can happen quite early in the boot process - and it took me some time
to get the order and also the environment variables right.  We should be able
to reuse that for the pure Guile modprobe.

After the patch we don't *manually* load any of the modules - and all the
tests work.  I booted my system with it - that also works.

P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked 
it?
glibc is like 5 kiB.  In that case saving 800 kiB is not really worth it...

Re: Kernel modules in initrd

[Ludovic Courtès]

Hello,

Danny Milosavljevic  skribis:

> I've got it to work now.  I've got a very minimal static kmod into the initrd 
> and
> that's now only loading modules for which supported hardware is present.
>
> On the other hand, the initrd got 800 kiB larger - I'm not sure why modprobe
> is so big... hmm...

Yeah.  If possible, I’d like to keep the initrd pure Scheme, to keep it
smallish.

I suppose the bits I’ve promised to detect necessary modules based on
modules.alias and /sys should be useful here?

Thanks,
Ludo’.

Re: Kernel modules in initrd

[Danny Milosavljevic]

Hi Andreas,

On Sat, 24 Feb 2018 00:02:39 +0100
Andreas Enge  wrote:

> On Fri, Feb 23, 2018 at 03:28:55PM +0100, Danny Milosavljevic wrote:
> > No, wait, according to 
> > https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723
> >  ,
> > the kernel should be doing that even without udev.  Are we sure we need to 
> > manually modprobe the stuff in gnu/build/linux-boot.scm in
> > the first place?  I think we should just add kmod to the initrd - that's 
> > it.  
> 
> in that case, if I understand correctly, the security question would not
> be a problem any more, right, as only really needed modules would be loaded
> by the kernel? Then we could add more modules to the initrd.

Yes.

I've got it to work now.  I've got a very minimal static kmod into the initrd 
and
that's now only loading modules for which supported hardware is present.

On the other hand, the initrd got 800 kiB larger - I'm not sure why modprobe
is so big... hmm...

Let's see the part Ludo adds to guix system init.  Maybe we can also replace
modprobe by it - if we want.

Re: Kernel modules in initrd

[ng0]

Ludovic Courtès transcribed 0.8K bytes:
> Hello,
> 
> Mark H Weaver  skribis:
> 
> > To make this easier, I think the right approach is to include many
> > modules like these to our installation image initrd, and then to
> > automatically detect which modules are needed for booting.  A future
> > easy installer could automatically add those modules to the OS config,
> > but in the meantime we could simply print a message recommending that
> > the user should add the needed modules to their initrd config.
> 
> FWIW I have code that I was planning to polish and submit (hopefully
> next week) that determines modules needed for a particular device by
> walking /sys.  In addition, I thought we could make the initrd module
> list a first class field of ‘operating-system’.  With these two pieces,
> ‘guix system’ should be able to warn when a module is missing.
> 
> Ludo’.

I'm willing to test those, it's probably less time consuming than figuring out
the corretc modules the Hetzner CLoud instance needs.
-- 
ng0 ::  https://n0.is | https://crash.cx
A88C8ADD129828D7EAC02E52E22F9BBFEE348588


signature.asc
Description: PGP signature

Re: Kernel modules in initrd

[Andreas Enge]

Hello,

On Fri, Feb 23, 2018 at 03:28:55PM +0100, Danny Milosavljevic wrote:
> No, wait, according to 
> https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723
>  ,
> the kernel should be doing that even without udev.  Are we sure we need to 
> manually modprobe the stuff in gnu/build/linux-boot.scm in
> the first place?  I think we should just add kmod to the initrd - that's it.

in that case, if I understand correctly, the security question would not
be a problem any more, right, as only really needed modules would be loaded
by the kernel? Then we could add more modules to the initrd.

Andreas

Re: Kernel modules in initrd

[Ludovic Courtès]

Hello,

Mark H Weaver  skribis:

> To make this easier, I think the right approach is to include many
> modules like these to our installation image initrd, and then to
> automatically detect which modules are needed for booting.  A future
> easy installer could automatically add those modules to the OS config,
> but in the meantime we could simply print a message recommending that
> the user should add the needed modules to their initrd config.

FWIW I have code that I was planning to polish and submit (hopefully
next week) that determines modules needed for a particular device by
walking /sys.  In addition, I thought we could make the initrd module
list a first class field of ‘operating-system’.  With these two pieces,
‘guix system’ should be able to warn when a module is missing.

Ludo’.

Re: Kernel modules in initrd

[Danny Milosavljevic]

No, wait, according to 
https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723
 ,
the kernel should be doing that even without udev.  Are we sure we need to 
manually modprobe the stuff in gnu/build/linux-boot.scm in
the first place?  I think we should just add kmod to the initrd - that's it.

Re: Kernel modules in initrd

[Danny Milosavljevic]

Hi Mark,

On Thu, 22 Feb 2018 17:01:11 -0500
Mark H Weaver  wrote:

> Every extra loaded kernel module means more RAM usage in the kernel, a
> larger initrd image that must be loaded by possibly slow bootloaders,
> and code complexity in the running kernel, leading to a greater attack
> surface for possible security exploits.  For example, last year some
> memory corruption bugs were found in the LSI MegaRAID SAS module.  See
> .

Good points.

> To make this easier, I think the right approach is to include many
> modules like these to our installation image initrd, and then to
> automatically detect which modules are needed for booting.  A future
> easy installer could automatically add those modules to the OS config,
> but in the meantime we could simply print a message recommending that
> the user should add the needed modules to their initrd config.

Good idea.

Another possibility that maybe would make the current situation less bad
would be to put udev-static into the initrd [1] and basically make it only
load the required modules on demand.

Also, udev uses a lot of tools like grep etc - we could use busybox to
get the size of the initrd down again then.

I definitely agree that we should only add modules possibly required for
early booting (until the rootfs is mounted) and not all the modules - that
would be insanely big.

On another note, let's please make the error detection and reporting better.

It should be easy to find unclaimed nodes by scanning /sys/class/pci_bus or
walking /sys/devices/pci:00, 
trying to find whether each has a "driver" entry and that would have caught
this problem and improved the diagnostics a lot.  This is not 1990 where
when we had no driver we didn't know the hardware was there.  We do know now.

[1] https://www.redhat.com/archives/fedora-devel-list/2004-May/msg01008.html

Re: Kernel modules in initrd

[Mark H Weaver]

Hi Andreas,

Andreas Enge  writes:

> recently I had an unpleasant experience in installing GuixSD: After booting
> with the USB key and following the installation instructions, then rebooting
> into the installed system, the "/" file system was not found: neither using
> file system labels, nor device nodes.
>
> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
> (apply base-initrd file-systems
>#:extra-modules '("mptbase" "mptsas" "mptscsih")
>rest)))
> to the operating-system declaration solved the problem.
> However, it took us quite some time and several trials to diagnose the
> problem in the first place.
>
> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>report that new ones are needed? For instance, the berlin server has
>modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

We need a better system for dealing with this.

I don't think that loading every kernel module that any GuixSD user ever
needed during the early boot process is a good approach.  To make
matters worse, it's easy for a user to add modules to their initrd, but
there's no easy way to remove modules from the default set.

I don't want *every* GuixSD user to unconditionally load, into their
kernels during early boot, code to support LSI MegaRAID SAS cards, just
because there exists one GuixSD user out there who needs it.

Every extra loaded kernel module means more RAM usage in the kernel, a
larger initrd image that must be loaded by possibly slow bootloaders,
and code complexity in the running kernel, leading to a greater attack
surface for possible security exploits.  For example, last year some
memory corruption bugs were found in the LSI MegaRAID SAS module.  See
.

To make this easier, I think the right approach is to include many
modules like these to our installation image initrd, and then to
automatically detect which modules are needed for booting.  A future
easy installer could automatically add those modules to the OS config,
but in the meantime we could simply print a message recommending that
the user should add the needed modules to their initrd config.

What do you think?

  Mark

Re: Kernel modules in initrd

[Andreas Enge]

Hello Danny,

On Thu, Feb 22, 2018 at 10:34:03PM +0100, Danny Milosavljevic wrote:
> When you booted from the USB key the system DIDN'T need mptsas to find
> ITS root filesystem.  So it booted fine and in the end of it all you had
> a working GNU Linux, with udev.
> 
> Then udev saw the hard disk and udev went and loaded the kernel
> module (from the root filesystem, not from the initrd), as it was
> supposed to do.

thanks for the explanations, I agree with your analysis.
So the patch you propose looks good to me.

Andreas

Re: Kernel modules in initrd

[Danny Milosavljevic]

Hi,

On Thu, 22 Feb 2018 22:29:25 +0100
Jan Nieuwenhuizen  wrote:

> (initrd (lambda (file-systems . rest)
>   (apply base-initrd file-systems
>  #:extra-modules '("pata_via" "pata_acpi" "sata_via")
>  rest)))

Hmm, see gnu/system/linux-initrd.scm linux-modules .

It should already have pata_acpi, pata_atiixp.  We probably should add 
pata_via, sata_via there...

> >modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

Sure, also to linux-modules.

Re: Kernel modules in initrd

[Danny Milosavljevic]

Hi Andreas,

On Thu, 22 Feb 2018 22:17:07 +0100
Andreas Enge  wrote:

> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
> (apply base-initrd file-systems
>#:extra-modules '("mptbase" "mptsas" "mptscsih")
>rest)))
> to the operating-system declaration solved the problem.

> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>report that new ones are needed? For instance, the berlin server has
>modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

Sure.

> 2) Better yet, since we managed to boot on the USB key and see the disk:
>Why not have the same modules in the initrd of the installation image
>and of the installed default system? 

I don't think that that is what happened.

The initrd is a very early root filesystem which is supposed to contain
enough stuff (programs and data) to be able to mount the actual root
filesystem.

When you booted from the USB key the system DIDN'T need mptsas to find
ITS root filesystem.  So it booted fine and in the end of it all you had
a working GNU Linux, with udev.

Then udev saw the hard disk and udev went and loaded the kernel
module (from the root filesystem, not from the initrd), as it was
supposed to do.

It's unfortunate that the initrd is needed, but let's just add all
maybe-required kernel modules to it.

Re: Kernel modules in initrd

[Jan Nieuwenhuizen]

Andreas Enge writes:

> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
> (apply base-initrd file-systems
>#:extra-modules '("mptbase" "mptsas" "mptscsih")
>rest)))
> to the operating-system declaration solved the problem.

Sounds familiar, when installing GuixSD on older hardware, I needed these

(initrd (lambda (file-systems . rest)
  (apply base-initrd file-systems
 #:extra-modules '("pata_via" "pata_acpi" "sata_via")
 rest)))

> However, it took us quite some time and several trials to diagnose the
> problem in the first place.

Yes.

> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>report that new ones are needed? For instance, the berlin server has
>modules (list "megaraid_sas" "libsas" "scsi_transport_sas").
> 2) Better yet, since we managed to boot on the USB key and see the disk:
>Why not have the same modules in the initrd of the installation image
>and of the installed default system? Not being able to see the disk
>in the beginning would at least have made the process fail early on,
>and moreover there seem to be more modules on the USB key. And what
>fits onto a 1 GB USB key should also easily fit on a hard disk...
>
> What do you think?

I'm not experienced in this, otoh if things "just work", that's always
nice.
janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com