Public bug reported:
# Ubuntu 26.04 on Raspberry Pi CM4: A/B kernel updates never apply
Two independent defects in the `piboot-try` A/B boot mechanism on Ubuntu 26.04
(resolute) for Raspberry Pi. Together they mean **no kernel update can ever be
applied** on an affected board: new kernels stage into `/boot/firmware/new/`,
are
marked `bad`, and the system keeps booting the original kernel indefinitely.
Both were found on a Raspberry Pi Compute Module 4 while provisioning a motion
controller. They are reported together because the first masks the second.
## Environment
| Item | Value |
|---|---|
| OS | Ubuntu 26.04.1 LTS (resolute), arm64 |
| Board | Raspberry Pi Compute Module 4 Rev 1.1 (eMMC, CM4 IO board) |
| Bootloader EEPROM build | 2022-04-26 (tryboot-capable) |
| `piboot-try` | 1.1ubuntu0.1 |
| `systemd` | 259.5-0ubuntu3.4 |
| `linux-image-raspi` | 7.0.0-1019.19 |
| Running kernel | 7.0.0-1017-raspi |
| `linux-firmware-raspi` | 14-0ubuntu1.1 |
| `rpi-eeprom` | 28.14-0ubuntu1 |
| Boot partition | 505 MB vfat, 290 MB free |
`/boot/firmware/autoboot.txt` contains `tryboot_a_b=1`; `config.txt` contains
`os_prefix=current/` under `[all]` and `os_prefix=new/` under `[tryboot]`
(stock, unmodified).
---
## Bug 1 — `piboot-try` uses a reboot form systemd 259 rejects
### Description
`/usr/share/flash-kernel/functions-piboot` triggers the trial boot with a
positional argument:
```sh
# functions-piboot:411
reboot "$boot_partition tryboot"
```
systemd 259 no longer accepts a positional argument to the `reboot` verb; it was
replaced by an option. From the systemd NEWS file shipped in this release:
> `"systemctl reboot" takes the option "--reboot-argument="`
On the affected system:
```console
$ systemctl --dry-run reboot "0 tryboot"
Too many arguments.
$ systemctl --dry-run reboot --reboot-argument="0 tryboot"
Failed to write reboot parameter file: Permission denied # parses; fails
only on privilege
```
### Effect
The `piboot-try-reboot` service logs `Rebooting to test new boot assets` and the
machine reboots, but **without** the tryboot argument reaching the firmware. The
next boot is an ordinary boot from `current/`, so `piboot-try` observes
`tryboot=0` while `new/state` is `trying`, concludes the trial failed, and
writes
`bad`:
```
piboot-try[693]: Marked new boot assets bad
```
`0/-/good/bad` is terminal — nothing retries the assets. The staged kernel is
never booted, and the user sees only the generic message:
```
New boot assets in /boot/firmware/new failed. Fallen back to known good state.
```
### Steps to reproduce
1. Install a kernel update on Ubuntu 26.04 for Raspberry Pi so assets stage into
`/boot/firmware/new/`.
2. `piboot-try --test` returns 0 (a double boot is pending).
3. Reboot.
4. Observe: still on the old kernel; `/boot/firmware/new/state` contains `bad`;
the journal shows `Marked new boot assets bad`.
Expected: the system double-boots, tests the staged assets, and promotes
them.
### Suggested fix
Use the supported form, with a fallback for older systemd:
```sh
if systemctl --version >/dev/null 2>&1; then
systemctl reboot --reboot-argument="$boot_partition tryboot"
else
reboot "$boot_partition tryboot"
fi
```
---
## Bug 2 — booting via `os_prefix=new/` fails, with a byte-identical
kernel
### Description
With Bug 1 worked around by hand, the tryboot **does** occur — and still fails.
Critically, it fails even when the staged assets are a byte-for-byte copy of the
assets that boot successfully from `current/`.
### Steps to reproduce
Starting from a working system:
```bash
# Make new/ hold exactly the kernel that current/ boots from
sudo cp /boot/firmware/current/vmlinuz /boot/firmware/current/initrd.img
/boot/firmware/new/
sudo cp /boot/firmware/current/*.dtb /boot/firmware/new/
printf 'trying\n' | sudo tee /boot/firmware/new/state
# Trigger the trial boot using the form systemd 259 accepts
sudo systemctl reboot --reboot-argument="0 tryboot"
```
Verified identical before rebooting:
```console
$ md5sum /boot/firmware/current/vmlinuz /boot/firmware/new/vmlinuz
f24e8e4a12f1e15e61d8842ed3a98e78 /boot/firmware/current/vmlinuz
f24e8e4a12f1e15e61d8842ed3a98e78 /boot/firmware/new/vmlinuz
```
### Actual behaviour
On an attached HDMI monitor: the rainbow splash appears (so `start4.elf` loaded
and ran), the kernel never starts, the board resets, and a second boot comes up
normally from `current/`. `new/state` is then `bad`.
The same kernel image boots without issue from `current/`.
### Expected behaviour
The trial boot should start the staged kernel, reach `multi-user.target`, and
`piboot-try-validate` should promote `new/` to `current/`.
### Ruled out
- **Corrupt or truncated assets.** `gzip -t` passes on `new/vmlinuz`; `zstd -t`
passes on `new/initrd.img`; the initrd size matches
`/boot/initrd.img-7.0.0-1019-raspi` exactly.
- **Missing assets.** `new/` is a strict superset of `current/`.
- **Boot partition full.** 290 MB free.
- **Malformed `cmdline.txt`.** `current/cmdline.txt` and `new/cmdline.txt` are
identical.
- **EEPROM too old for tryboot.** Bootloader build 2022-04-26, well after
tryboot
support landed.
- **Kernel lacking tryboot support.** The string `tryboot` is present in both
kernel images.
- **A bad 1019 kernel.** The failure reproduces with the 1017 kernel that boots
fine from `current/`.
### Remaining lead (not yet tested)
After the experiment above, the only structural difference between the two slots
is that `new/` contains 17 GPU firmware files that `current/` does not:
```
bootcode.bin fixup.dat fixup4.dat fixup4cd.dat fixup4db.dat fixup4x.dat
fixup_cd.dat fixup_db.dat fixup_x.dat start.elf start4.elf start4cd.elf
start4db.elf start4x.elf start_cd.elf start_db.elf start_x.elf
```
These are byte-identical to the copies at the root of the boot partition
(`start4.elf` md5 `37ce15a4f86e3c063a6cef29d62aa1da` in both places).
`find_bootloader_assets` in `functions-piboot` (lines 82–84) copies them into
the
slot deliberately, while `current/` lacks them because it was produced by the
migration from the older flat layout.
This asymmetry is worth investigating: booting with `os_prefix=new/` causes the
firmware to load `start*.elf` from the prefixed directory, a path that a normal
boot from `current/` never exercises. Deleting the firmware blobs from `new/`,
so
that the slot matches `current/`, would confirm or eliminate this.
`new/overlays` also holds 371 files against `current/`'s 370, though both
contain
the overlays required for this board.
---
## Impact
- No kernel update can be applied. Security updates to `linux-raspi` install
into
`/boot/firmware/new/` and are never booted. `uname -r` stays pinned to
whatever
kernel first shipped on the image.
- Switching kernel flavours is impossible — this was found while trying to adopt
`linux-image-raspi-realtime` (PREEMPT_RT) for a robotics workload.
- The user-facing message names no cause, and `piboot-try --reset-new` simply
repeats the failure, so the condition looks like faulty hardware.
## Workarounds
Neither is complete: the first fixes only Bug 1, and Bug 2 then blocks
the boot.
```bash
# Clear the terminal "bad" state
sudo piboot-try --reset-new
# Drive the trial boot with the argument form systemd 259 accepts
printf 'trying\n' | sudo tee /boot/firmware/new/state
sudo systemctl reboot --reboot-argument="0 tryboot"
```
## Attachments worth collecting
- `journalctl -b -u piboot-try-reboot -u piboot-try-validate`
- `/boot/firmware/config.txt`, `/boot/firmware/autoboot.txt`
- `ls -la /boot/firmware/current /boot/firmware/new`
- `cat /boot/firmware/{current,new}/state`
- `od -An -tu4 --endian=big /proc/device-tree/chosen/bootloader/tryboot`
> Note: timestamps in early-boot journal entries on a CM4 are misleading. The
> board has no RTC, so entries before chrony syncs carry a stale date. Messages
> that appear to be weeks old are frequently from the current boot.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: piboot-try 1.1ubuntu0.1
ProcVersionSignature: Ubuntu 7.0.0-1017.17-raspi 7.0.12
Uname: Linux 7.0.0-1017-raspi aarch64
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: arm64
CasperMD5CheckResult: unknown
CloudArchitecture: aarch64
CloudID: nocloud
CloudName: unknown
CloudPlatform: nocloud
CloudSubPlatform: config-disk (/dev/mmcblk0p1)
Date: Sat Sep 19 11:41:53 2026
ImageMediaBuild: 20260826
SourcePackage: piboot-try
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: piboot-try (Ubuntu)
Importance: Undecided
Status: New
** Tags: apport-bug arm64 arm64-image raspi-image resolute
** Attachment added: "piboot-try-diagnostics.txt"
https://bugs.launchpad.net/bugs/2167775/+attachment/6001249/+files/piboot-try-diagnostics.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2167775
Title:
Ubuntu 26.04 on Raspberry Pi CM4: A/B kernel updates never apply
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/piboot-try/+bug/2167775/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
