Hi Don, There is no reason I have to limit to 8 queues. Since I am struggling to get BW up to expect, I have tried different queue number but not able to see good result.
As far as interrupt go, I did verify driver use msi-x. We are running single app on top of minimum Centos (no UI, etc), so CPU is not an issue here. Attached is is verbose version of pci info, again, thanks! Hank On Wed, Sep 7, 2016 at 10:28 AM, Skidmore, Donald C < donald.c.skidm...@intel.com> wrote: > Just out of curiosity why are you limiting yourself to 8 queue? The > drivers default behavior would be to allocate one queue pre CPU (up to 63) > and use ATR to attempt to isolate the flows to the CPU the application is > running on. Even if you don’t like ATR with your data flow you could use > the full 16 queue that RSS supports. > > > > Also the script that was supplied with the driver was to be used with ATR > in mind it assumes that you have one vector per CPU, that might be causing > you some difficultly as well. > > > > I can’t see much with the lspci output you really need to turn on the > verbose option (lspci -vvv to useful information. > > > > Thanks, > > -Don > > > > *From:* Hank Liu [mailto:hank.tz...@gmail.com] > *Sent:* Wednesday, September 07, 2016 10:20 AM > *To:* Skidmore, Donald C <donald.c.skidm...@intel.com> > *Cc:* e1000-devel@lists.sourceforge.net > *Subject:* Re: [E1000-devel] Intel 82599 AXX10GBNIAIOM cards for 10G SFPs > UDP performance issue > > > > Thanks for quick response and helping. I guess I didn't make it clear is > that the application (receiver, sender) open 240 connections each > connection has 34 Mbps traffic. I have tried default 40 Rx queues (40 logic > CPUs), 8 Rx queues. I also looked into /proc/interrupts, I can see > interrupt came in core I expected (e.g. with irq balance on, interrupt > spreads on all 40 logic CPUs; with irq balance off, I use script came with > Intel driver tar file to pin to certain cores (e.g. 0-3, 0-7), I can see > interrupt spreads accordingly. I also use /proc/net/softnet_stat to see if > rx poll in driver still have job to do. Yes, it is even I > increase net.core.netdev_budget value. > > > > Below is pci info, Thanks again and hopefully I can get some guide here... > > > > Hank > > > > 00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DMI2 (rev 01) > > 00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D PCI Express Root Port 1 (rev 01) > > 00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D PCI Express Root Port 2 (rev 01) > > 00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D PCI Express Root Port 2 (rev 01) > > 00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D PCI Express Root Port 3 (rev 01) > > 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) > > 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) > > 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) > > 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) > > 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) > > 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) > > 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) > > 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) > > 00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Map/VTd_Misc/System Management (rev 01) > > 00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D IIO Hot Plug (rev 01) > > 00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01) > > 00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O > APIC (rev 01) > > 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset > SPSR (rev 05) > > 00:11.1 SMBus: Intel Corporation C610/X99 series chipset MS SMBus 0 (rev > 05) > > 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA > Controller [AHCI mode] (rev 05) > > 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI > Host Controller (rev 05) > > 00:16.0 Communication controller: Intel Corporation C610/X99 series > chipset MEI Controller #1 (rev 05) > > 00:16.1 Communication controller: Intel Corporation C610/X99 series > chipset MEI Controller #2 (rev 05) > > 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB > Enhanced Host Controller #2 (rev 05) > > 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express > Root Port #1 (rev d5) > > 00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express > Root Port #4 (rev d5) > > 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB > Enhanced Host Controller #1 (rev 05) > > 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC > Controller (rev 05) > > 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port > SATA Controller [AHCI mode] (rev 05) > > 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller > (rev 05) > > 02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > > 02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller > 10-Gigabit X540-AT2 (rev 01) > > 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller > 10-Gigabit X540-AT2 (rev 01) > > 08:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA > G200e [Pilot] ServerEngines (SEP1) (rev 05) > > 7f:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 0 (rev 01) > > 7f:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D QPI Link 0 (rev 01) > > 7f:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 0 (rev 01) > > 7f:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 1 (rev 01) > > 7f:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D QPI Link 1 (rev 01) > > 7f:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 1 (rev 01) > > 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R3 QPI Link 0/1 (rev 01) > > 7f:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > > 7f:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > > 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R3 QPI Link Debug (rev 01) > > 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > 7f:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R2PCIe Agent (rev 01) > > 7f:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R2PCIe Agent (rev 01) > > 7f:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Ubox (rev 01) > > 7f:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D Ubox (rev 01) > > 7f:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Ubox (rev 01) > > 7f:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Home Agent 0 (rev 01) > > 7f:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D Home Agent 0 (rev 01) > > 7f:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01) > > 7f:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01) > > 7f:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > 7f:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > 7f:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > 7f:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > 7f:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01) > > 7f:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Global Broadcast (rev 01) > > 7f:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 01) > > 7f:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 01) > > 7f:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01) > > 7f:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01) > > 7f:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > 7f:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > 7f:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > 7f:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > 7f:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 01) > > 7f:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 01) > > 7f:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 01) > > 7f:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 01) > > 7f:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Target Address/Thermal/RAS (rev 01) > > 7f:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01) > > 7f:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Global Broadcast (rev 01) > > 7f:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control (rev 01) > > 7f:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > 7f:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > 7f:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > 7f:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > 7f:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 7f:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > 80:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) > > 80:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) > > 80:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) > > 80:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) > > 80:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) > > 80:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) > > 80:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) > > 80:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) > > 80:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Map/VTd_Misc/System Management (rev 01) > > 80:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D IIO Hot Plug (rev 01) > > 80:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01) > > 80:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O > APIC (rev 01) > > ff:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 0 (rev 01) > > ff:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D QPI Link 0 (rev 01) > > ff:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 0 (rev 01) > > ff:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 1 (rev 01) > > ff:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D QPI Link 1 (rev 01) > > ff:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D QPI Link 1 (rev 01) > > ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R3 QPI Link 0/1 (rev 01) > > ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > > ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > > ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R3 QPI Link Debug (rev 01) > > ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Caching Agent (rev 01) > > ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D R2PCIe Agent (rev 01) > > ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D R2PCIe Agent (rev 01) > > ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Ubox (rev 01) > > ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D Ubox (rev 01) > > ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Ubox (rev 01) > > ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Home Agent 0 (rev 01) > > ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > E3 v4/Xeon D Home Agent 0 (rev 01) > > ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01) > > ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01) > > ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01) > > ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01) > > ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Global Broadcast (rev 01) > > ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 01) > > ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 01) > > ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01) > > ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01) > > ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > > ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 01) > > ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 01) > > ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 01) > > ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 01) > > ff:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Target Address/Thermal/RAS (rev 01) > > ff:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01) > > ff:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Global Broadcast (rev 01) > > ff:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control (rev 01) > > ff:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > ff:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > ff:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > ff:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > > ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > v4/Xeon D Power Control Unit (rev 01) > > > > > > On Wed, Sep 7, 2016 at 9:54 AM, Skidmore, Donald C < > donald.c.skidm...@intel.com> wrote: > > > -----Original Message----- > > From: Hank Liu [mailto:hank.tz...@gmail.com] > > Sent: Wednesday, September 07, 2016 9:37 AM > > To: e1000-devel@lists.sourceforge.net > > Subject: [E1000-devel] Intel 82599 AXX10GBNIAIOM cards for 10G SFPs UDP > > performance issue > > > > I run into 10G card performance issue. Here is my test setup, > > > > . optical cable p2p between sender and receiver machines on eth2 port. > > . both ends have Intel 82599 card with 4.x (4.0, 4.3.5, 4.4.6) ixgbe > linux driver . > > host machines are 2 sockets E5 Intel server with 64G memory on board . > OS is > > Centos 7 build 1511. > > . UDP multicast traffic sent from sender to receiver. Receiver has very > little > > send traffic . Packet payload size is 1316 or 1372. No small packet. > > > > Problem: Seeing a lot flow control packets sent from receiver end to > sender, > > hence performance drop when bw above 7 Gbps. If flow control is tuned > off, > > seeing a lot of rx_no_dma_resources. Note: Sender has no problem to > catch up > > BW request. > > > > Tune has been done in Rx side are, > > . IRQ balance on / off > > . RSC queue # > > . Rx ring size - up to 4096 > > . rx_back_log size > > . interrupt modulation > > . pin worker threads on right side CPUs (same socket) > > > > I was not able to make the BW go higher than 8 Gpbs reliably (no packet > drop > > or slow down). Any suggestion? Thanks! > > > > > > Hank > > If I had to guess what is happening it would be that all your traffic is > going to one queue and thus you're work load is only being processed by one > CPU. This would be easy to verify by looking that your interrupt > distribution in /proc/interrupts both before and after your test. > > It would also be interesting to check for anything out of the ordinary in > the system log as well as the bus layout (lspci -vvv). > > Thanks, > -Don Skidmore <donald.c.skidm...@intel.com> > > >
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
