I have an Intel 82599 10 gigabit ethernet card I wanted to get working
on my SiFive Unmatched board.
I found the ix(4) driver has some weirdness around MSI-X
interrupts. While the driver supports operating both with and without
MSI-X support, it's hard-coded via a flag rather than dynamically checking
if it's available. If the flag is set (which it always is right now),
but MSI-X isn't available, the driver will throw an error and the device
won't work:
ix0 at pci7 dev 0 function 0 "Intel 82599" rev 0x01ixgbe_allocate_msix:
pci_intr_map_msix vec 0 failed
The root cause is this call failing in if_ix.c:
if (pci_intr_map_msix(pa, i, &ih)) {
printf("ixgbe_allocate_msix: "
"pci_intr_map_msix vec %d failed\n", i);
error = ENOMEM;
goto fail;
}
Because in _pci_intr_map_msix (in sys/arch/riscv64/dev/pci_machdep.c):
if ((pa->pa_flags & PCI_FLAGS_MSI_ENABLED) == 0 ||
pci_get_capability(pc, tag, PCI_CAP_MSI, NULL, NULL) == 0)
return -1;
The PCI attach flags would not have PCI_FLAGS_MSI_ENABLED set.
The following diff remedies that by checking if PCI_FLAGS_MSI_ENABLED is
actually set, rather than just trying and failing because the hard-coded
flag says so. It also enables ix(4) in the kernel config for
riscv64. Effectively, the driver will now only try to use MSI-X if the
machine is advertising it to be available.
I've tested this on riscv64 (obviously) and an amd64 machine with the
same model card.
On riscv64 the device shows up as:
ix0 at pci7 dev 0 function 0 "Intel 82599" rev 0x01: intx, address
90:e2:ba:29:18:72
And seems to be happily passing packets.
On amd64, the device still shows up as using MSI-X as it did before:
ix0 at pci2 dev 0 function 0 "Intel 82599" rev 0x01, msix, 8 queues,
address 90:e2:ba:29:76:a0
I notice in some of the other drivers for Intel ethernet chipsets, this
same flag exists. I don't know if it's preferable to make this same
change to them nor do I have the hardware to test such a change, but if
this is something that's of interest I'd be happy to do it.
Thanks!
Index: sys/arch/riscv64/conf/GENERIC
===================================================================
RCS file: /cvs/src/sys/arch/riscv64/conf/GENERIC,v
retrieving revision 1.26
diff -u -p -u -p -r1.26 GENERIC
--- sys/arch/riscv64/conf/GENERIC 12 Jul 2021 19:11:42 -0000 1.26
+++ sys/arch/riscv64/conf/GENERIC 19 Jul 2021 22:53:38 -0000
@@ -94,6 +94,7 @@ wsdisplay* at radeondrm?
em* at pci? # Intel Pro/1000 Ethernet
bge* at pci? # Broadcom BCM57xx (aka Tigon3)
oce* at pci? # Emulex OneConnect 10Gb ethernet
+ix* at pci? # Intel 82598EB 10Gb ethernet
nvme* at pci? # NVMe controllers
ahci* at pci? # AHCI SATA controllers
Index: sys/dev/pci/if_ix.c
===================================================================
RCS file: /cvs/src/sys/dev/pci/if_ix.c,v
retrieving revision 1.178
diff -u -p -u -p -r1.178 if_ix.c
--- sys/dev/pci/if_ix.c 22 Dec 2020 23:25:37 -0000 1.178
+++ sys/dev/pci/if_ix.c 19 Jul 2021 22:53:38 -0000
@@ -202,7 +202,6 @@ struct cfattach ix_ca = {
};
int ixgbe_smart_speed = ixgbe_smart_speed_on;
-int ixgbe_enable_msix = 1;
/*********************************************************************
* Device identification routine
@@ -1780,7 +1779,7 @@ ixgbe_setup_msix(struct ix_softc *sc)
int nmsix;
unsigned int maxq;
- if (!ixgbe_enable_msix)
+ if (!(pa->pa_flags & PCI_FLAGS_MSI_ENABLED))
return;
nmsix = pci_intr_msix_count(pa->pa_pc, pa->pa_tag);
