A likely candidate for this, reproduced on bare metal (noble, linux-
image-7.0.0-38-generic, QEMU 1:8.2.2+ds-0ubuntu1.18, AMD EPYC with AMD-
Vi in translated mode, Samsung PM1725 NVMe). It explains the cache=none
vs writeback behaviour and the "only on kernel 7.0" part, but I could
not make a plain GRUB boot fail with it, so whether it is what hits your
VMs still needs confirming on your hosts (see the end).
**Candidate cause**
A Linux 7.0 block-layer regression, fixed upstream by 96c8ea3c5add
"block: save page offset gaps in cloned bio" (Fixes: 2f6b2565d43c, which
is in v7.0). When a bio gets split, the front clone loses its segment-
gap bit. nvme-pci then coalesces the request into a single IOVA range;
the second segment ends up on a non-page-aligned IOVA, the IOMMU mapping
fails and the I/O returns EIO. Neither noble linux-hwe-7.0 7.0.0-38 nor
7.0.0-39 (proposed) has the fix yet.
It needs all of these at once:
- NVMe. Only nvme-pci uses this DMA path, so hosts with SATA/SAS storage never
see it.
- No NVMe multipath head device. On dual-port (CMIC) drives the kernel creates
a path device such as `nvme0c0n1` underneath `nvme0n1`; the head splits the bio
first and the per-path queue recomputes the gap bit, which hides the bug.
Single-port drives, which only have `nvme0n1`, or any drive with
`nvme_core.multipath=0`, are exposed.
- IOMMU in translated mode, i.e. no `iommu=pt`.
- An O_DIRECT (`cache=none`) request with sub-page segment boundaries that is
big enough to be split (more than max_sectors_kb or max_segments).
`cache=writeback` goes through the page cache, whose segments are page aligned,
so it is immune. That is why the workaround works.
Note: on Ubuntu 7.0.0-38 the host log does not contain "iommu:
unaligned" (this path fails in `iommu_map_nosync()`, which does not
log). The host-side signature would be:
I/O error, dev nvme0n1, sector 1859393536 op 0x1:(WRITE) flags
0xc800 phys_seg 33 prio class 2
**Reproducer**
Attached `lp2169551-repro.sh` is a single file:
./lp2169551-repro.sh check # can this host hit it?
./lp2169551-repro.sh host # O_DIRECT pwritev with page-offset gaps:
10/10 EIO on an affected host
./lp2169551-repro.sh vm # noble cloud image, cache=none, same test
inside the guest: I/O error, dev vda
CACHE=writeback ./lp2169551-repro.sh vm # control: passes
Results on one host and one kernel, toggling only the multipath head:
| multipath head | host test | guest test, cache=none | guest test,
cache=writeback |
|---|---|---|---|
| present (default) | 20/20 OK | 30 plain boots OK, 12 guests x 10 min fio OK |
OK |
| disabled | 10/10 EIO | 10/10 EIO, `I/O error, dev vda` | 10/10 OK |
bpftrace on a failing request: `bio_split clone gap_bit=0` -> request
`phys_gap_bit=0 segs=33` -> `dma_iova_try_alloc` succeeds -> second
`dma_iova_link` at offset 4608 -> `iommu_map_nosync` returns -6. Plain
boots and contiguous-buffer workloads did not fail here because their
virtio requests have page-aligned segment boundaries; if this is the bug
you are seeing, whether a given VM hits it would depend on the guest I/O
pattern and on the NVMe's max_sectors_kb / max_segments, which would
explain the randomness.
**Workarounds** (if this is it): GA 6.8 kernel, `disk-
cachemodes="file=writeback"`, or `iommu=pt` on the host.
**Next steps**
Giuseppe, to confirm or rule this out, could you check whether an
affected compute host matches the conditions above? Mainly:
```
ls /sys/block | grep nvme # only nvme0n1 (single-port), or
also a path device like nvme0c0n1 (dual-port)?
cat /sys/class/nvme/nvme0/model # NVMe model
cat /proc/cmdline # any iommu=pt?
journalctl -k | grep 'I/O error, dev nvme' # at the time a VM dropped into
GRUB rescue? (if none, it is probably not this)
```
** Attachment added: "lp2169551-repro.sh"
https://bugs.launchpad.net/charm-nova-compute/+bug/2169551/+attachment/6012040/+files/lp2169551-repro.sh
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169551
Title:
VMs on ephemeral fails to boot [arm64, amd64]
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-compute/+bug/2169551/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
