The latest ixgbe driver from e1000.sourceforge.net has the set_irq_affinity.sh script included. I would certainly suggest you use a newer driver because we're constantly improving the driver for performance and fixing bugs we've found. The version you have is a couple of years old.
As far as your questions regarding improving your RX performance, there's no quick answer. Just running the latest driver and affinity script is the only general answer I can give you and everything else is usually specific to what you're doing. If you have specific questions, I'd suggest contacting your Intel representative and we can discuss things. Todd Fujinaka Software Applications Engineer Networking Division (ND) Intel Corporation [email protected] (503) 712-4565 -----Original Message----- From: Brian Fallik [mailto:[email protected]] Sent: Monday, January 28, 2013 6:46 AM To: [email protected] Subject: [E1000-devel] igxbe rx throughout - IRQs cs CPUs Hi, I'm trying to diagnose a UDP receive throughput bottleneck using the igxbe driver and the 82599EB NIC. Under ingress load we're seeing packet drops at the NIC (via `watch cat ifconfig eth4`). Digging deeper, I noticed that /proc/interrupts reported that only CPU0 was servicing IRQs generated by the NIC. By manually adjusting CPU affinity in /proq/irq/*/smp_affinity I was able to spread the load across more cores to avoid the packet drops. See below for some basic info about our platform. My first question is if this is the recommended approach to improve Rx throughput? Is there are some driver setting or script (e.g. http://code.google.com/p/ntzc/source/browse/trunk/zc/ixgbe/set_irq_affinity.sh?r=16) that handles this automatically or should we be manually assigning queues to CPUs? Also, should we assign each queue to a different CPU or use a subset of CPUs? How should I consider processors, cores, and hyperthreads when considering this assignment? And, lastly, would this be addressed if we upgraded to a more modern driver? Thanks, brian === platform info === # cat /proc/version Linux version 2.6.32-71.el6.x86_64 ([email protected]) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Fri May 20 03:51:51 BST 2011 # cat /etc/issue CentOS Linux release 6.0 (Final) Kernel \r on an \m # ethtool -i eth4 driver: ixgbe version: 2.0.62-k2 firmware-version: 0.9-3 bus-info: 0000:05:00.0 # lspci -v -v -s 05:00.1 05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 50 Region 0: Memory at df380000 (64-bit, non-prefetchable) [size=512K] Region 2: I/O ports at dce0 [size=32] Region 4: Memory at df2fc000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at d0000000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB Capabilities: [e0] Vital Product Data Unknown small resource type 00, will not decode more. Capabilities: [100] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-d7-c9-5c Capabilities: [150] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [160] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 01 VF offset: 128, stride: 2, Device ID: 10ed Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable) Region 3: Memory at 0000000000000000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Kernel driver in use: ixgbe Kernel modules: ixgbe ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnnow-d2d _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
