On Fri, 8 Oct 2010, Chris Friesen wrote:
> > Hi all, > > We've got an 82599 device using the XAUI interface over an ATCA > backplane data link, using version 2.0.72.4 of the ixgbe driver from > sourceforge. (We also tried 2.1.4 and it didn't help.) > > As I mentioned a while back, I'm investigating an issue where under load > we eventually suddenly stop receiving packets. We're sending out > packets, I can see the reply being sent with tcpdump on other hosts, but > we never receive the reply. The driver doesn't complain about loss of > carrier (and doesn't indicate dropped packets). > > I added some instrumentation that triggers once we haven't received any > packets for at least 300ms (and I know that we should be receiving > some). The instrumentation dumps the values of various > interrupt-related registers on the 82599 and also walks the rx rings to > see if there are any packets ready to process. I guess, first did you end up with the wmb/rmb patches added by IBM for the PPC stuff upstream? I'm not sure we have those in our sf.net driver yet. > Here's the result of a typical event: > > eth4 (affected device) > EICS: 00000001 > EIMS: d619fffe > EIMC: d619fffe > EIAC: 4000ffff > EIAM: 0000ffff > packets waiting on rx ring 0 (leading zeros added above for clarity) so this says you have an interrupt on vector zero, which is masked, which means if you read the RDH/RDT registers you would probably see they are farther than two apart (meaning data was rx'd by hardware into memory but not taken by driver) > For comparison, here is what eth3 looks like. This device is also an > 82599, part of the same bond link, but isn't showing any problems. > EICS: 00000000 > EIMS: d619ffff > EIMC: d619ffff > EIAC: 4000ffff > EIAM: 0000ffff > no packets waiting on any rx rings > Given the above, it appears that we're somehow getting into a state > where the interrupts are automatically disabled but then the driver > doesn't re-enable them right away. Eventually it does re-enable them, > but we've seen it take up to 750ms for traffic to start flowing again. I think that delay you see is probably time before the watchdog "kicks" the interrupt, worst case up to two seconds. > We've got a few clues--it seems to be related to tcp traffic, it doesn't > take much traffic at all (less than 10% of a single core cpu usage), and > dropping the link speed from 10Gbps to 1Gbps made the problem go away. so dropping the link speed changes timing and probably eliminates your issue via luck. > Anyone got any ideas what might be going on? To me this sounds extremely like an interrupt enable race, possibly at the hardware level with your chipset that is implementing MSI-X One thing we can do is stop using autoclear (EIAC), which requires a slight bit of code to be added to the MSI-X routines, to make sure the driver writes EICR to clear the bit that was asserted. > Chris > > PS The manual says that EICS and EIMC are both write-only, yet the > driver reads both of them for the register dump and reads EICS for > actual operation. Is this using undocumented behaviour? I believe that the read of EICS may be undocumented, but our parts have always had this behavior (back to pci-x e1000) reading EIMC is documented for something (maybe 1GbE parts) as a way to see the mask ------------------------------------------------------------------------------ Beautiful is writing same markup. Internet Explorer 9 supports standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. Spend less time writing and rewriting code and more time creating great experiences on the web. Be a part of the beta today. http://p.sf.net/sfu/beautyoftheweb _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
