On Tue, Feb 14, 2012 at 10:52:49AM +0100, David Imhoff wrote:
> >Synopsis: Intel PRO/1000 PF em() network card not working with MSI
> >on Dell R610
> >Category: Kernel/i386
> >Environment:
> System : OpenBSD 5.1
> Details : OpenBSD 5.1 (GENERIC.MP) #5: Mon Feb 13 12:12:18 CET
> 2012
>
> [email protected]:/usr/src/sys/arch/i386/compile/GENERIC.MP
>
> Architecture: OpenBSD.i386
> Machine : i386
> >Description:
> My Intel PRO/1000 PF network card in a Dell PowerEdge R610 server,
> stopped
> functioning after upgrading to OpenBSD 5.0. The card does not
> transmit any
> data. When trying to send a packet the following error appears in the
> kernel logging:
> "em0: watchdog timeout -- resetting"
>
> I found that disabling MSI in the em* driver, by always setting
> 'sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED;' in em_attach(),
> made the card function again. Of course this is not a structural fix.
>
> The problem does not seem card or driver related. Since the same model
> and subtype card, in a Dell PowerEdge 1950, does function properly
> with
> MSI enabled. I also tried the same setup with multiple revisions of
> the
> R610, with the same result.
This appears to be a silicon level flaw:
>From "Intel 82571EB/82572EI Ethernet Controller Specification Update":
63. Byte Enables 2 and 3 Are Not Set on MSI Writes Problem: MSI
(format code definition Message Signal Interrupts) writes on the 82571EB
will not have the upper two Byte Enables (BE) set.
Implication: The PCI specification requires Byte Enables 2 and 3 to be
set even though that data will always be zero. Because the
82571/82572 does not set these Byte Enables, MSI writes fail to
generate interrupts on systems with chipsets that have been designed
to require these Bytes Enables to be set. This errata only applies
when MSI is supported and enabled by the system and OS.
Workaround: None, As long as MSI is being used, Byte Enables 2 and 3
will not be set.
Status: No Fix.
--
So it seems to be safe we should not try to enable MSI on 82571/82572
at all.
Index: if_em.c
===================================================================
RCS file: /cvs/src/sys/dev/pci/if_em.c,v
retrieving revision 1.261
diff -u -p -r1.261 if_em.c
--- if_em.c 5 Oct 2011 02:52:09 -0000 1.261
+++ if_em.c 14 Feb 2012 10:27:03 -0000
@@ -329,8 +329,11 @@ em_attach(struct device *parent, struct
/* Determine hardware revision */
em_identify_hardware(sc);
- /* Only use MSI on the newer PCIe parts */
- if (sc->hw.mac_type < em_82571)
+ /*
+ * Only use MSI on the newer PCIe parts, with the exception
+ * of 82571/82572 due to "Byte Enables 2 and 3 Are Not Set" errata
+ */
+ if (sc->hw.mac_type <= em_82572)
sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED;
/* Parameters (to be read from user) */
