The virtio-net function of the AMD Pensando DSC2-200 (Elba) DPU, as
shipped by Oracle (1af4:1041, subsystem 108e:1200), cannot DMA to
arbitrary 64-bit bus addresses, and Oracle advises limiting it to 48
bits. virtio_pci gives it a 64-bit mask, so its virtqueues stop
completing requests once the IOMMU starts handing it IOVAs above 2^48.
The card is used in OCI bare-metal shapes such as BM.Standard.E6.288.
On those AMD EPYC 9005 hosts the AMD IOMMU supports a 6-level v1 page
table (EFR[HATS] = 2), so the aperture spans all 64 bits and the IOVA
allocator, which works down from min(mask, aperture), returns addresses
just below 2^64:
- With iommu.forcedac=1, the first control virtqueue command never
completes and virtio_net hangs in probe.
- Without it, iommu_dma_alloc_iova() prefers 32-bit IOVAs until that
space runs out once, logs "Using 64-bit DMA addresses", and stops
preferring them for the device. Across several thousand of these
hosts, 21 logged that for the virtio-net function within 0.2 to 7.3
days of boot. On 14 of them, TX queues then stopped completing
descriptors within hours ("NETDEV WATCHDOG: ... transmit queue N
timed out"). No host without the fallback reported a TX timeout.
virtio gives a device no way to advertise a DMA address limit, and the
platform describes none for this device, so the driver has to apply
one. Cap the device at 48 bits through the dma_mask override that
vp_modern_probe() already provides for devices with address limits.
The device still gets 32-bit IOVAs first, and then IOVAs below 2^48
instead of 2^64. pds_vdpa uses 52 bits for the same ASIC; this uses
Oracle's narrower 48-bit limit, which still leaves 256 TiB of IOVA
space.
Only the virtio-net function is matched, since it is the one that was
tested. force_legacy does not bypass the quirk: the legacy probe
refuses device IDs above 0x103f, so the device always goes through the
modern probe.
Tested on a BM.Standard.E6.288 with 6-hour iperf3 --bidir soaks, at up
to 57 Gbit/s each way with iommu.forcedac=1 and at about 26 Gbit/s each
way without it. Neither run saw a TX timeout or an AMD-Vi
IO_PAGE_FAULT. With forcedac, every traced IOVA for the device fell
between 2^32 and 2^48; without it, every IOVA stayed below 2^32, as
before.
Cc: [email protected] # 6.5+
Co-developed-by: Kyle McMartin <[email protected]>
Signed-off-by: Kyle McMartin <[email protected]>
Signed-off-by: Vlad Poenaru <[email protected]>
---
drivers/virtio/virtio_pci_common.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/virtio/virtio_pci_common.c
b/drivers/virtio/virtio_pci_common.c
index 79d3395f37050..0a7c97cd80509 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -667,8 +667,18 @@ static const struct dev_pm_ops virtio_pci_pm_ops = {
#endif
+/*
+ * The virtio-net function of the AMD Pensando DSC2-200 (Elba) DPU, as
+ * shipped by Oracle (subsystem 108e:1200), cannot DMA to arbitrary 64-bit
+ * bus addresses. Oracle advises limiting it to 48 bits.
+ */
+#define VIRTIO_PCI_QUIRK_DMA_48BIT BIT(0)
+
/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
static const struct pci_device_id virtio_pci_id_table[] = {
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_REDHAT_QUMRANET, 0x1041,
+ PCI_VENDOR_ID_SUN, 0x1200),
+ .driver_data = VIRTIO_PCI_QUIRK_DMA_48BIT },
{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
{ 0 }
};
@@ -710,6 +720,9 @@ static int virtio_pci_probe(struct pci_dev *pci_dev,
if (rc)
goto err_enable_device;
+ if (id->driver_data & VIRTIO_PCI_QUIRK_DMA_48BIT)
+ vp_dev->mdev.dma_mask = DMA_BIT_MASK(48);
+
if (force_legacy) {
rc = virtio_pci_legacy_probe(vp_dev);
/* Also try modern mode if we can't map BAR0 (no IO space). */
--
2.53.0-Meta
