Public bug reported:
I updated my DL360 gen8 server to the newest kernel and got a non
functioning P420i due to massive errors in dmesg. Since I'm not that
familiar with kernel stuff and storage drivers I've asked an LLM to
provide all necessary information to file a bug against the ubuntu
kernel package. This is the result of the information that was already
found. Let me know if you need any further information on this issue.
# [regression] 6.8.0-134: IOVA allocator hands out addresses inside BIOS
RMRR regions, causing DMA mapping failures and storage controller loss
(HP ProLiant Gen8 / P420i)
**Package:** linux (Ubuntu Noble 24.04)
**Broken:** 6.8.0-134.134 (6.8.12)
**Last good:** 6.8.0-124.124
**Also present in:** 6.8.0-136.136 (per changelog; no fix/revert present)
**Severity:** total storage loss on affected hosts — 32 TB RAID volume dropped
offline, 10 production VMs lost their disks
---
> **Note on how this report was produced.** The diagnosis, log analysis and the
> text of this report were assembled with the help of an AI assistant (Claude)
> working on the affected machine under my supervision. I have reviewed it and
> can vouch for the underlying facts; the raw logs are attached so every claim
> can be checked independently. The address/RMRR correlation below was verified
> programmatically against the raw log rather than by eye.
>
> **I am happy to supply any additional information, run diagnostics, or test a
> proposed kernel on request** — see the "Control machine" note near the end.
---
## Summary
On 6.8.0-134, the Intel IOMMU IOVA allocator hands out IOVAs that fall inside
BIOS-declared RMRR regions. Those regions are already identity-mapped, so
`__domain_mapping()` finds an existing PTE, throws `WARNING at
drivers/iommu/intel/iommu.c:2227`, and the DMA mapping fails:
```
WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227
__domain_mapping+0x2b5/0x320
DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003)
```
Note `bdf81003` — vPFN 0xbdf81 is mapped to PFN 0xbdf81, i.e. an identity
mapping, which is what RMRR regions get.
This is **not driver-specific**. On this host it hit three drivers within 90
seconds: `nvme` first, then `hpsa`, then `tg3`. The HP Smart Array P420i was
simply the one that could not recover: its logical volume reset failed, the
device went offline (`DID_NO_CONNECT`), and the 32 TB RAID-5 volume backing our
LVM volume group disappeared, taking 10 production VMs with it.
## Root cause evidence
The BIOS declares these RMRR regions (identical on both kernels, see
below):
```
DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff -> PFN 0xbdf6e
DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff -> PFN 0xbdf6f-0xbdf7e
DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff -> PFN 0xbdf7f-0xbdf82
DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff -> PFN 0xbdf83-0xbdf84
union -> PFN 0xbdf6e-0xbdf84
```
The complete set of colliding vPFNs observed across all 66 errors:
```
0xbdf6e 0xbdf6f 0xbdf70 0xbdf71 0xbdf72 0xbdf73 0xbdf74 0xbdf75
0xbdf76 0xbdf77 0xbdf78 0xbdf79 0xbdf7a 0xbdf7b 0xbdf7c 0xbdf7d
0xbdf7e 0xbdf7f 0xbdf80 0xbdf81 0xbdf82 0xbdf83 0xbdf84
```
That is `0xbdf6e`-`0xbdf84` — an exact, complete match with the RMRR union.
All 23 distinct colliding IOVAs lie inside an RMRR region; none lies outside
one. (Checked programmatically over the attached log, not by inspection.) The
RMRR reserved-region exclusion in the IOVA allocator appears not to be in
effect.
## This is not a firmware or BIOS problem
The RMRR regions declared by the BIOS are **byte-for-byte identical** on
6.8.0-124 and 6.8.0-134 (diffed; both logs attached). Same machine, same BIOS
(P71, 05/24/2019), same controller firmware. The only variable is the kernel.
6.8.0-124 handles these exact regions correctly.
## This is not passthrough-specific
No VFIO/PCI passthrough is configured on this host (`lsmod | grep vfio` -> 0).
This is a plain KVM/libvirt host using virtio and LVM-backed disks. Reports
elsewhere tie the 6.8.12 IOMMU regression to VFIO group assignment; this
reproducer has no VFIO involvement at all, which suggests the underlying defect
is broader than the passthrough symptom it was first noticed through.
## A/B evidence (same physical machine, consecutive boots)
| Boot | Kernel | Uptime | DMAR errors | I/O errors |
|------|--------|--------|-------------|------------|
| -2 | 6.8.0-124 | 2026-06-13 -> 2026-07-14 (31 days) | 0 | 0 |
| -1 | 6.8.0-134 | ~10 min to controller loss | 66 | 4701 |
| 0 | 6.8.0-124 | since revert, stable | 0 | 0 |
Booting 6.8.0-124 restored full function immediately: controller status OK,
volume group back, all VMs recovered.
## Timeline (boot -1, kernel 6.8.0-134)
```
12:32:34 boot
12:41:24 first WARNING - nvme_map_data (mdraid rebuild driving heavy DMA)
12:42:46 hpsa_scatter_gather hits the same warning
12:42:50 tg3_start_xmit hits it too (network path)
12:43:23 hpsa: resetting logical Direct-Access HP LOGICAL VOLUME RAID-5
12:44:32 hpsa: reset logical failed / sd 3:1:0:0: rejecting I/O to offline
device
sda offline -> volume group gone -> 10 VMs lose their disks
13:25:05 reboot into 6.8.0-124 -> fully recovered
```
The first failure came ~9 minutes in, under sustained DMA load from an mdraid
rebuild — consistent with the allocator only reaching the RMRR range once lower
IOVA space was under pressure. Hosts may therefore appear fine after boot and
fail later under load.
## Representative stack trace
```
WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227
__domain_mapping+0x2b5/0x320
CPU: 32 PID: 562 Comm: kworker/32:1H Tainted: G I
6.8.0-134-generic #134-Ubuntu
Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019
Workqueue: kblockd blk_mq_run_work_fn
RIP: 0010:__domain_mapping+0x2b5/0x320
Call Trace:
<TASK>
intel_iommu_map_pages+0xe1/0x140
__iommu_map+0x121/0x280
iommu_map_sg+0xbf/0x1f0
iommu_dma_map_sg+0x463/0x4f0
__dma_map_sg_attrs+0x35/0xd0
dma_map_sgtable+0x1d/0x40
nvme_map_data+0xd6/0x480 [nvme]
nvme_prep_rq.part.0+0x35/0x120 [nvme]
nvme_queue_rq+0x7e/0x207 [nvme]
blk_mq_dispatch_rq_list+0x137/0x520
__blk_mq_sched_dispatch_requests+0x118/0x190
blk_mq_sched_dispatch_requests+0x2f/0x80
blk_mq_run_work_fn+0x72/0x90
process_one_work+0x184/0x3a0
worker_thread+0x18b/0x330
kthread+0xf2/0x120
ret_from_fork+0x47/0x70
ret_from_fork_asm+0x1b/0x30
</TASK>
---[ end trace 0000000000000000 ]---
DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003)
```
The same warning with `hpsa_scatter_gather+0x3e/0x310 [hpsa]` and with
`__tg3_start_xmit+0x1f0/0xd60 [tg3]` in place of the nvme frames is in the
attached log (46 nvme, 18 hpsa, 4 tg3 occurrences).
## Environment
```
Hardware: HP ProLiant DL360p Gen8
BIOS: HP P71, 05/24/2019
CPU: Intel Xeon E5-2697 v2 @ 2.70GHz (48 threads)
RAM: 503 GB
Controller: Smart Array P420i [103c:323b] rev 01, subsystem [103c:3354]
PCI 0000:02:00.0, IOMMU group 49
Driver: hpsa 3.4.20-200
OS: Ubuntu 24.04.4 LTS
Cmdline: BOOT_IMAGE=/boot/vmlinuz-6.8.0-134-generic root=UUID=<redacted> ro
maybe-ubiquity
(no intel_iommu= or iommu= parameter set; IOMMU on by default, 132
groups)
Workload: KVM/libvirt host, ~30 VMs, LVM on hpsa RAID-5 + LVM on NVMe mdraid1
```
**Control machine:** a second, identical DL360p Gen8 (same BIOS, same P420i)
running 6.8.0-101 has 18+ weeks uptime with zero DMAR errors. It has not been
exposed to -134 and is available to test a proposed kernel. I can also
reproduce on the primary host in a maintenance window if that is more useful.
## Related reports
Same signature (`DMA PTE ... already set` from a RAID driver's scatter-gather
mapping, leading to controller resets) reported downstream on Proxmox, where a
Proxmox developer states "the -29 one had an IOMMU related regression that is
fixed there", referring to their 6.8.12-30 kernel:
-
https://forum.proxmox.com/threads/has-anyone-seen-dma-pte-megaraid-controller-resets-after-upgrading-to-recent-proxmox-8-kernels.184389/
-
https://forum.proxmox.com/threads/proxmox-kernel-6-8-12-29-causing-array-controller-errors.184181/
Proxmox 6.8.12-29/-30 share the 6.8.12 base with Ubuntu's 6.8.0-134. If the fix
there is a revert of an upstream stable commit, the same revert would plausibly
apply here. **This is an inference from the shared code base and matching
signature, not something I was able to confirm** — the referenced upstream
report (kernel bugzilla #221234) is behind an anti-bot gate, and I could not
identify the offending commit.
An older, still-untriaged Ubuntu bug with the same signature:
https://bugs.launchpad.net/bugs/1970453
## Workarounds
- Boot 6.8.0-124 (what we did; fully effective)
- `iommu=pt` or `intel_iommu=off` reported effective by others (untested here)
None of these are acceptable long-term for hosts that must stay on a supported
kernel.
## Request
Please identify and revert/backport the fix for the RMRR reserved-region
handling in the 6.8.0 Noble series.
## Attachments and how they were prepared
The attached logs have been **filtered before upload** — please read this so you
know what you are and are not looking at:
- `[UFW BLOCK]` firewall lines were **removed entirely**. They carried
third-party
source IP addresses (personal data under GDPR) and were unrelated to the
fault.
- Hostname, IP addresses, MAC addresses and UUIDs were replaced with
placeholders.
- Full `dmidecode` output is **not attached**: it embeds the chassis serial
number
three times over (including as a raw hex dump), and the system UUID is a hex
encoding of that same serial. `hardware-summary.txt` contains the model-level
fields instead.
No kernel-relevant line was altered. The evidence was counted before and after
filtering to confirm nothing was lost: 66 DMAR errors, 66 `iommu.c:2227`
warnings, 132 `__domain_mapping` frames, 23 distinct vPFNs, 9 RMRR regions and
the complete hpsa reset-to-offline chain are all present in both. **If you need
an unfiltered log, ask me and I will provide one through a private channel.**
- `dmesg-6.8.0-134-BAD.log` — full kernel log of the failing boot
- `dmesg-6.8.0-124-GOOD-boot.log` — boot log of the working kernel, showing the
identical RMRR regions and no collisions
- `lspci-vvnn.txt` — full PCI topology
- `hardware-summary.txt` — curated hardware/firmware details
- `version.txt` — kernel, cmdline, distro, hpsa module info
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: iommu kernel-bug noble regression-update
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2160616
Title:
[regression] 6.8.0-134: IOVA allocator hands out addresses inside BIOS
RMRR regions, causing DMA mapping failures and storage controller loss
(HP ProLiant Gen8 / P420i)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2160616/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
