Public bug reported:
SUMMARY
After a routine security update installed linux-image-6.8.0-136-generic
(6.8.0-136.136) alongside the previously-running linux-
image-6.8.0-117-generic (6.8.0-117.117), a cold boot into 6.8.0-136 hung
mid-boot. Both mdadm RAID1 arrays spanning two identical NVMe boot
drives assembled in a degraded 1-of-2-mirror state, and the system's
network never came up (unreachable at the OS level; console showed no
further progress for several minutes past the hang point). A subsequent
boot into the immediately-prior kernel, 6.8.0-117-generic, on the exact
same hardware, same boot media, same RAID members, came up completely
clean with both arrays at [2/2] [UU].
This strongly suggests a regression between 6.8.0-117 and 6.8.0-136
affecting NVMe device enumeration/probe timing (one of the two RAID1
members not being detected in time during array assembly), rather than a
hardware fault -- the same physical drives, controllers, and PCIe slots
are proven working under 6.8.0-117.
ENVIRONMENT
- Ubuntu 24.04.4 LTS (noble)
- uname -a (known-good kernel currently in use): Linux flash 6.8.0-117-generic
#117-Ubuntu SMP PREEMPT_DYNAMIC Tue May 5 19:26:24 UTC 2026 x86_64 x86_64
x86_64 GNU/Linux
- Affected package: linux-image-6.8.0-136-generic 6.8.0-136.136 (from
noble-updates/noble-security)
- Working package: linux-image-6.8.0-117-generic 6.8.0-117.117
- CPU: AMD EPYC 9135, 16-core (SP5, Family 1Ah Model 02h)
- Motherboard: ASRockRack GENOAD24QM32-2L2T/BCM (BMC: AST2600, MegaRAC SP
firmware 10.17.00)
- System: ASRockRack 1U24E1S-GENOA/2L2T
- RAM: 128GB, 4800 MT/s
- Affected boot RAID members: 2x Intel Optane NVMe SSD P1600X Series (PCI IDs
8086:2525), providing /boot (md0, RAID1) and / via an LVM-on-md1 RAID1 mirror
- 22 additional data-path NVMe drives (Samsung 990 PRO, unaffected -- these are
on a separate ZFS pool, not part of the mdadm boot arrays)
STEPS TO REPRODUCE
1. On this hardware, with linux-image-6.8.0-136-generic installed and set as
the default GRUB boot target, perform a full cold boot (not a warm reboot --
this hardware has a separate known PCIe link-training sensitivity on warm
resets, so testing was done via a full poweroff + BMC-triggered power-on to
rule that out as a confound).
2. Observe kernel boot log via KVM/console.
OBSERVED (6.8.0-136, failing)
Boot log via KVM (transcribed from console; a full serial/dmesg capture
of this exact failing boot was not available at the time -- Serial-over-
LAN was being brought up in parallel during this same incident):
[ 19.332063] Btrfs loaded, zoned=yes, fsverity=yes
[ 50.584308] md/raid1:md0: active with 1 out of 2 mirrors
[ 50.590443] md0: detected capacity change from 0 to 4188160
[ 50.599179] md/raid1:md1: active with 1 out of 2 mirrors
[ 50.605326] md1: detected capacity change from 0 to 83818496
No further kernel log lines appeared for several minutes (system
confirmed unresponsive: no SSH/network reachability, no response to
keypresses on console, verified via a second independent KVM/console
session ruling out a display-only artifact). Note the ~31-second gap
between the "Btrfs loaded" line at T+19.3s and the RAID1 assembly at
T+50.6s -- consistent with one NVMe controller enumerating very late or
timing out during probe.
EXPECTED / OBSERVED-GOOD (6.8.0-117)
Same hardware, same physical drives, cold boot into 6.8.0-117-generic
instead:
$ cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 nvme13n1p3[3] nvme12n1p3[2]
41909248 blocks super 1.2 [2/2] [UU]
md0 : active raid1 nvme13n1p2[3] nvme12n1p2[2]
2094080 blocks super 1.2 [2/2] [UU]
unused devices: <none>
Both arrays fully assembled, [2/2] [UU], system boots normally, network
and all services come up.
SUSPECTED ROOT CAUSE / RELATED UPSTREAM WORK
Possibly related to the upstream PCIe "Configuration Request Retry
Status" (CRS) polling rework (d591f6804e7e / e259e3258f3f, "PCI: Wait
for device readiness with Configuration RRS"), which changes how the
kernel handles slow-to-respond PCIe endpoints during enumeration. This
mechanism is discussed in Launchpad bug #2111521
(https://bugs.launchpad.net/bugs/2111521) in the context of Intel
datacenter NVMe drives failing detection at boot starting around
6.8.0-60 -- mechanistically consistent with the symptom here (a slower-
enumerating NVMe controller, in this case an Optane P1600X, not being
ready in time for mdadm assembly). Whether this specific commit is
present in 6.8.0-136 but not 6.8.0-117, and whether it's the actual
cause here, has not been confirmed -- flagging as the most plausible
lead found so far, not a confirmed diagnosis.
WORKAROUND
Pinned GRUB default to boot 6.8.0-117-generic instead of the newer
6.8.0-136-generic until this is understood/resolved. linux-
image-6.8.0-136-generic remains installed for testing but is not the
default boot target.
ADDITIONAL NOTES
- This is a homelab server, not customer-facing, so I can boot 6.8.0-136 again
on request to capture better diagnostics (e.g., a full serial-console dmesg of
the actual hang, which was not available for the original incident) if that
would help triage.
- Full apport diagnostic data (collected via "apport-cli -f -p linux
--save=..." while running the known-good 6.8.0-117 kernel) is attached:
CurrentDmesg, Lspci/Lspci-vt, ProcCpuinfo, ProcKernelCmdLine, ProcModules,
ProcInterrupts, UdevDb, dmi.*, Dependencies, RelatedPackageVersions, etc.
- Note on redaction: drive serial numbers and WWN identifiers in the attached
UdevDb have been replaced with REDACTED-<8-hex-digit-hash>-REDACTED
placeholders (a deterministic hash of the original value). The same physical
device always maps to the same placeholder everywhere it appears (by-id
symlinks, ID_SERIAL_SHORT, ID_WWN, etc.), so cross-referencing which entries
refer to the same drive is still possible -- only the literal serial/WWN values
themselves are hidden. Model, revision, PCI bus path, and all other fields are
untouched.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: boot mdadm nvme raid regression
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161494
Title:
linux-image-6.8.0-136-generic hangs at boot with degraded mdadm RAID1
(1/2 mirrors) on dual NVMe boot array; 6.8.0-117-generic boots clean
on identical hardware
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2161494/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
