On Fri, 8 Oct 2010, Chris Friesen wrote:
> On 10/08/2010 11:14 AM, Chris Friesen wrote: > > > Given the above, it appears that we're somehow getting into a state > > where the interrupts are automatically disabled but then the driver > > doesn't re-enable them right away. Eventually it does re-enable them, > > but we've seen it take up to 750ms for traffic to start flowing again. > > > > We've got a few clues--it seems to be related to tcp traffic, it doesn't > > take much traffic at all (less than 10% of a single core cpu usage), and > > dropping the link speed from 10Gbps to 1Gbps made the problem go away. > > > > Anyone got any ideas what might be going on? > > I just realized there are a few more pieces of information that might be > useful. > > The system has two quad-core cpus, each with hyperthreading, so linux > sees 16 cpus. > > /proc/interrupts shows that each device has 16 interrupts with names of > the form ethX-TxRx-Y where X and Y vary appropriately, as well as one > interrupt of the form ethX:lsc. Currently all interrupts are handled by > cpu0. (I realize this isn't necessarily optimal, but it's how it is > currently.) depending on the kernel version you have, you might be overrunning the irq stack with all irqs from our adapters on one core, you may not notice it but it might be causing random panics that are extremely difficult to debug (just in case you're seeing it). The patch that went upstream serializes all hard irq handlers on a cpu to solve the stack smash problem. But, thats not the current issue. > ethtool -S shows that each device has 16 tx queues and 16 rx queues. do you happen to be running with rx-usecs = 0/InterruptThrottleRate set? I'm going to reply to your patch email now. ------------------------------------------------------------------------------ Beautiful is writing same markup. Internet Explorer 9 supports standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. Spend less time writing and rewriting code and more time creating great experiences on the web. Be a part of the beta today. http://p.sf.net/sfu/beautyoftheweb _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired