https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296662
Bug ID: 296662
Summary: bhyve: passthru caches PCI config header while device
is still recovering from FLR; physical BARs left
zeroed
Product: Base System
Version: 15.1-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: bhyve
Assignee: [email protected]
Reporter: [email protected]
Created attachment 272687
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272687&action=edit
Proof of concept, not for commit: readiness wait + BAR restore in cfginit()
### Environment
- Host: FreeBSD 15.1-RELEASE (releng/15.1-n283562-96841ea08dcf) GENERIC
amd64
- CPU: AMD Ryzen (Cezanne), AMD-Vi IOMMU (ivhd0), hw.vmm.amdvi.enable=1
- Device: Intel Arc A380 (DG2), 8086:56a5 rev 05, subvendor 1849:6006
(ASRock), plus its HDA audio function 8086:4f92; both attached to
ppt(4) via pptdevs="3/0/0 4/0/0"
- Guest: Ubuntu 24.04.4, kernel 6.17.0-35-generic, booted via grub-bhyve
(vm-bhyve 1.7.3), devices: -s 0:6:0,passthru,3/0/0
-s 0:7:0,passthru,4/0/0
- IOMMU initializes fine (hw.vmm.iommu.initialized: 1); the EN-25:20.vmm
fix is included in 15.1.
### Overview
Intel DG2 GPUs do not complete FLR cleanly: the Transactions Pending bit
never clears, and after the FLR the device does not respond to config
space accesses (all reads return 0xff) for up to ~1 second. Measured on
this card with `devctl reset pci0:3:0:0`: reads return 0xffffffff
immediately after the reset and recover within ~1 s.
When such a device is assigned to a VM, two things go wrong in sequence:
1. ppt(4) resets the device (ppt_pci_reset() -> pcie_flr(dev, ...,
force); "pci0:3:0:0: Transactions pending after FLR!" is logged on
every start). pcie_flr() waits a fixed delay and then
pci_cfg_restore() writes the saved config into a device that is not
yet responding, so the restore is lost: afterwards the physical BARs
read back as zero (only the type flag bits remain, e.g. 0x00000004 /
0x0000000c) and PCIM_CMD_MEMEN / PCIM_CMD_BUSMASTEREN are clear in
the command register.
2. Immediately afterwards, bhyve's passthru_init() -> cfginit() copies
the physical PCI header into the emulated config space:
for (int i = 0; i <= PCIR_MAXLAT; i += 4)
pci_set_cfgdata32(pi, i, passthru_read_config(...));
Because the device is still unresponsive, vendor/device are cached
as 0xffff. Since the standard header range is served to the guest
from this cache ("Emulate most PCI header register",
set_pcir_handler(sc, 0, PCIR_MAXLAT + 1, passthru_cfgread_emulate,
...)), the guest permanently sees an empty slot. A PCI rescan in the
guest does not help even after the device has recovered, because the
poisoned cache is never refreshed.
Fast-resetting functions are unaffected — the audio function 4f92 of
the same card resets instantly and is always visible in the guest —
which is what makes the failure look so puzzling: one function of the
card appears, the other silently does not, with no error from bhyve.
Even when the header is read correctly (see PoC below), the guest
driver then fails with "Device is non-operational; MMIO access returns
0xFFFFFFFF!" because of the zeroed physical BARs from step 1.
### Steps to Reproduce
1. Attach an Intel Arc GPU function to ppt(4) via pptdevs.
2. Start any bhyve VM with `-s N,passthru,B/S/F` for that function.
3. Observe "Transactions pending after FLR!" on the host console.
4. In the guest: `lspci` — the slot is empty.
`echo 1 > /sys/bus/pci/rescan` does not help.
5. On the host, `pciconf -r pciB:S:F 0x10` etc. show zeroed BARs and a
cleared command register, while `pciconf -r ... 0x00` returns the
correct vendor/device ID (the card itself has recovered).
### Actual Results
Guest sees no device; bhyve reports no error. After manually restoring
the BAR values and command register with pciconf(8) on the host and
re-scanning the PCI bus in the guest, the i915 driver initializes the
GPU completely (GuC/HuC load and authenticate, /dev/dri/renderD128
works, VA-API transcoding functional) — confirming that nothing else is
missing.
### Expected Results
The passthru device is visible and functional in the guest, or at
minimum bhyve reports a diagnostic that the device did not respond.
### Root cause confirmation (proof of concept)
To confirm the analysis, a proof-of-concept change to cfginit() was
tested on the affected system (see attachment):
- poll PCIR_VENDOR (50 ms x 40, i.e. up to 2 s) until the device
responds before caching the config space header;
- then compare each BAR register of the physical device against the
kernel's cached value (PCIOCGETBAR, which survives the reset),
rewrite it if it was lost, and re-enable memory decoding and bus
mastering.
With the PoC applied, a plain `vm stop` / `vm start` cycle produces a
fully working GPU in the guest with no manual intervention; bhyve logs
"PCI 3/0/0: restoring BAR0 after reset" / "...BAR2..." on each start.
This confirms both halves of the analysis (poisoned header cache; lost
config restore). The PoC is provided for reproduction and as a
mitigation for affected users — not as a ready-to-commit fix.
Suggested directions for a proper fix (left to the maintainers):
- Kernel: pcie_flr() in sys/dev/pci/pci.c could poll for device
readiness (vendor ID != 0xffff, as Linux does in pci_dev_wait())
after issuing the FLR and before pci_cfg_restore(). That would fix
the lost-restore problem for all consumers of pcie_flr(), not just
bhyve, and is probably the right layer for it.
- bhyve: cfginit() should not cache a config header read from a device
that is still returning 0xff, and could verify/restore the physical
BARs against the kernel's cached values before use.
### Observed behaviour with and without the PoC
- Without the PoC: on every VM start, "Transactions pending after FLR!"
on the host; guest sees an empty slot; physical BARs read back zeroed
(0x00000004 / 0x0000000c) with MEMEN/BUSMASTEREN cleared; manually
rewriting the BARs and command register with pciconf(8) plus a PCI
rescan in the guest brings the device fully up — confirming the
mechanism.
- With the PoC: plain `vm stop` / `vm start` cycles produce a fully
functional GPU in the guest with no manual intervention. In the
Ubuntu 24.04 guest (kernel 6.17), i915 initializes completely (DMC,
GuC 70.36.0 loaded, HuC authenticated), /dev/dri/renderD128 is
created, and VA-API reports full decode/encode profiles
(H.264/HEVC/VP9/AV1); hardware transcoding is functional.
- The audio function (8086:4f92) of the same card, which resets
quickly, behaves identically before and after the PoC — the readiness
wait is effectively a no-op for devices that recover promptly, and
the BAR restore writes nothing when registers already match the
kernel's cached values.
usr.sbin/bhyve/pci_passthru.c is currently identical in main and
releng/15.1, so the PoC applies unmodified to both; all testing was
done on 15.1-RELEASE.
### Provenance disclosure
The root-cause analysis was performed interactively on the affected
hardware; all measurements, register dumps and logs in this report are
from that system. The analysis and the attached proof-of-concept diff
were developed with the assistance of an LLM (Anthropic Claude). I am
not a C developer and have not performed a full code review of the
diff, so I am explicitly NOT proposing it for commit; it is attached
only to document how the root cause was confirmed and as a temporary
mitigation for affected users. Please treat it as a reproduction aid —
the root cause description should be sufficient for a committer to
implement a proper fix independently.
What I can vouch for personally is the empirical side: the reproduction
steps, the register dumps, the before/after behaviour, and the fact
that the described mechanism (manual BAR restore via pciconf(8) + guest
PCI rescan) brings the device fully up without any code changes. I am
happy to test any proposed fix on the affected hardware and can provide
any additional diagnostics on request.
A related but independent bug (bhyve's gvt-d probe claiming discrete
Intel GPUs and failing VM startup) is filed separately.
--
You are receiving this mail because:
You are the assignee for the bug.
