Hi, Background ========== We currently need to have images images to easily install Guix on ARM computers.
On most ARM computers, there is no boot flash that holds the
BIOS/UEFI. Instead most ARM CPU/SOC have builtin read-only code
(called bootrom) that can load a bootloader from a storage device (like
a microSD, eMMC, etc).
The bootloder is most of the time device specific. There are some rare
cases like u-boot-am335x-evm that can handle a family of computers, but
to make that possible they need to be built specifically for that.
For u-boot-am335x-evm, all the compatible computers all use same serial
port (like /dev/ttyS0), have the eMMC wired to the same pins, have a
tiny flash chip (not enough to hold the bootloader) with the board name
inside, etc. And this looks rather the exception than the norm.
All the diversity combined with the boot mechanism requires to have an
image per bootloader package at least.
Without that, users need to build their own image and this is
extremely error prone as you need to know:
- The offsets at which the various bootloader pieces need to be
installed.
- Exactly how GPT or MBR look on the disk.
- In some cases, what the bootrom requires (for instance it might
support only MBR and require you to set the bootable flag on the
partition with the bootloader).
- In some cases the serial port speed (like 1500000 on some boards and
115200 on others) and path (like /dev/ttyS0, or /dev/ttyMXC0).
Because of all that it is possible to build images that don't work or
that can boot but cannot boot anymore after a guix system reconfigure,
or that even corrupts the filesystem.
The problem is that adding many images don't scale. I thought that
this was a problem that would show up later once Guix system really
become usable on ARM, but the reality is that the problem is already
there. According to vagrantc on Codeberg[1]:
> I am somewhat reluctant to keep building images that are all in the
> GB scale, that share 99% with the other images built for the same
> architecture. That seems pretty rough on CI.
Short term solution
===================
The immediate action I propose here is to add as many images as we
want/need (like 1 per device), potentially having functions that
generate these images like for u-boot when they are really similar, and
to not disable substitutes for them.
And instead we would only build very few of them (or none) on official
Guix builders (bordeaux.guix.gnu.org, berlin.guix.gnu.org).
This way it would allow users and/or other builders to build these and
produce substitutes.
For instance if I find the time I'd also like to ask a French
non-profit (Libre en Communs) for a VM on a KGPE-D16 specifically to
build substitutes for these images.
In case you also need a better path forward before taking a decision
like that, I've also added a longer term plan below.
Longer term plan
================
On the official builders we could build only 4 images for ARM. To do
that we could reserve space for u-boot in the images and find a way to
later add u-boot to these images in a separate step.
This could be done with instructions, with a script generated by Guix,
or directly in Guix somehow (Guix would just use substitutes for
the generic image and the u-boot binary as inputs to assemble the
final image for the user), or with a combination of these: if the
script is generated, instructions could be written to use it, and Guix
could also run it, this way it covers both users with guix installed
and users without.
The idea here is to not require an eMMC with a hardware boot partition
(mmcblkNboot{0,1}) or a separate storage device to be able to run these
images on real hardware, though that would also work (it probably
already works).
The 4 ARM images could be:
- Two with GRUB for ARM UEFI: one 32bit, one 64bit. This would require
u-boot to then load GRUB (this should already work on some
ARM computers).
- Two without u-boot (32bit and 64bit) but with the
/boot/extlinux/extlinux.conf that is required by u-boot to boot.
We could also extend that to x86 somehow and have 5 images:
- Two with BIOS (32 and 64bit)
- Two with UEFI (32 and 64bit)
- One with UEFI 32bit and a 64bit OS.
How to get there ?
==================
All this has implications that might require a bit of work:
- On 64bit ARM, '(kernel linux-libre-arm64-generic)' results in a
computer that boots but the computer is not usable in many
cases.
For instance linux-libre-arm64-generic doesn't have WireGuard, and
it probably lack other configuration required by distributions.
Though here the above doesn't conflict with solving this issue
either by having generic kernels that include WireGuard etc, or by
adding more and more modules in a generic image.
- And if people do need different u-boot configuration to be able to
run GRUB (like UEFI support in u-boot to run GRUB), they could also
try to contribute that directly in u-boot first. I've done that for
the TBS2910 a few years ago, so u-boot probably still accept that
kind of patches.
Though there are some harder issues:
- The serial port speed and path (like /dev/ttyS0 or /dev/ttyMXC0)
varies between computers.
Maybe there is something in UEFI to somehow retrieve the serial
ports. In any case with the devicetree we do have access to the
computer name.
Here's an example on the rockpro64:
root@rockpro64 ~# strings \
/sys/firmware/devicetree/base/compatible
pine64,rockpro64-v2.1
pine64,rockpro64
rockchip,rk3399
So in the worst case we can have a database of device <-> serial
port settings and have a service use that. This could be shipped in
a generic image.
Though might also be possible to retrieve these from the device tree
as well (it requires upstream support for that):
[root@lime2 ~]# strings \
/sys/firmware/devicetree/base/chosen/stdout-path
serial0:115200n8
[root@lime2 ~]# strings \
/sys/firmware/devicetree/base/aliases/serial0
/soc/serial@1c28000
[root@lime2 ~]# strings \
/sys/firmware/devicetree/base/soc/serial@1c28000/status
okay
[root@lime2 ~]# strings \
/sys/firmware/devicetree/base/soc/serial@1c28000/compatible
snps,dw-apb-uart
And then 'snps,dw-apb-uart' identify a driver (here
drivers/tty/serial/8250/8250_dw.c) that we can link to
/dev/ttyS<number> through Kconfig (the option help often mention
the path) and/or the driver code. In some cases the path (ttyS vs
ttyO) is dependent on the kernel configuration (like
CONFIG_SERIAL_8250_OMAP_TTYO_FIXUP). The <number> comes from serial0.
I've not checked if with UEFI on ARM we still had the devicetree, but
in the meantime we can also have images without UEFI anyway as we
need u-boot in both cases when there is no boot flash.
I've also not looked yet into ACPI (ACPI seems to have a SPCR
(Serial Port Console Redirection) table for that).
The issue here is mostly the path (like /dev/ttyS0). For the speed
there are also other ways to handle it:
- One way would be to add a 'keep option for the (baud-rate [...])
of agetty-service-type. All the rest can be handled upstream:
u-boot typically enables serial, in linux, this can also be
configured in the devicetree, and Linux can even keep the current
speed (usually set by u-boot).
- Another option could be to standardize the speed on 115200. We
could also ship the computer-specific u-boot documentation with
the serial port speed changed with (substitute* [...]).
Finally we would need a place to list the names of the computers that
are supposed to work (for instance for a given image), and the manual
looks a good place for that. We might also want to list what works
(specific image, installer, etc).
Goals
=====
The long term plan could just be defined and/or implemented step by
step as we go. The goal would just to make images as generic as
possible, and also to have something that works and that is as clear
as possible for users (so we also need the manual to have a list of
computers that have full support: if there is only the bootloader in
Guix, that's insufficient for instance).
Installing an image is easy, running commands is also relatively easy
if that actually works.
In my case I have access to several ARM 32bit and 64bit computers that
run u-boot. Some only boot through microSD, other have boot flash or
eMMC. And I know my way around that.
I've also several Coreboot compatible computers, so I can also look
into ACPI if needed, though I'm way more familiar with ARM booting
than x86, and I also don't have lot of experience with UEFI on x86.
And I'm also pretty new to scheme/guile, so I might need help to
properly implement services to make images more generic.
This could be done step by step over multiple years though.
References:
-----------
[1]https://codeberg.org/guix/guix/pulls/5608
pgpngK7C_dsoC.pgp
Description: OpenPGP digital signature
