Flow director (ATR) use a different hash then RSS. When it creates a hash bucket for a flow that hash stays around in memory on the NIC until the table overfills or the part is reset (i.e. link up/down). For example in your ATR case you filled all hash buckets twice, meaning we had to flush the table twice and repopulated them when new packets arrive. This isn't uncommon for system with large number of connections that change rapidly and two table flushes is not really much of an issue. However large numbers of table flushes can be a sign that you data flow isn't a good candidate for ATR. Which is why we included this counter in the ethtool stats.
Thanks, -Don Skidmore <donald.c.skidm...@intel.com> > -----Original Message----- > From: Michał Purzyński [mailto:michalpurzyns...@gmail.com] > Sent: Thursday, October 06, 2016 2:04 PM > To: Skidmore, Donald C <donald.c.skidm...@intel.com> > Cc: Łukasz Chrustek <luk...@chrustek.net>; Tal Abudi > <talab...@gmail.com>; e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] rx queue > > Out of curiosity - does it use flow director for that, like creating a > short(ish) > living rule or there's another way? > > > On 6 Oct 2016, at 21:32, Skidmore, Donald C > <donald.c.skidm...@intel.com> wrote: > > > > Hey Lukasz, > > > > ATR work by targeting TCP flows as the adapter transmits a SYN packet. At > this point it creates a new hash bucket with the intent of redirecting other > traffic of this flow to the same queue was transmitted on. The idea being > this > what the received packets will arrive at the same CPU as the transmitted. > Clearly there is lots of traffic this wouldn't work well with (i.e. most > packet > generators, short lived connections, UPD). So if no hash is found for Rx > traffic > we default to RSS, which as I mentioned mux's over only the first 16 queues > due to the hash limitation. > > > > I believe this is what you are seeing. This is reinforced by the number > packets that are missing the ATR hash: > > > > fdir_match: 107665723 > > fdir_miss: 31074999856 > > > > Thanks, > > -Don Skidmore <donald.c.skidm...@intel.com> > > > > > >> -----Original Message----- > >> From: Łukasz Chrustek [mailto:luk...@chrustek.net] > >> Sent: Thursday, October 06, 2016 12:09 PM > >> To: Skidmore, Donald C <donald.c.skidm...@intel.com>; Tal Abudi > >> <talab...@gmail.com> > >> Cc: e1000-devel@lists.sourceforge.net > >> Subject: Re: [E1000-devel] rx queue > >> > >> Hello, > >> > >> I didn't change any default config, so if ATR is turned on by > >> default then it is on. Flows - this was generated flood traffic from > >> command > >> hping3 -2 --flod --rand-source <target IP>. <target IP> is > >> behind problematic server - the server is router. Can You tell what > >> is the way to verify this by ethtool ? > >> > >> Regards > >> Lukasz > >> > >>> It looks like your just using RSS to spread most of your traffic > >>> between queues who's hash is limited to 4 bits (16 queues). This > >>> would happen if you turned off ATR or your flows weren't applicably > >>> for ATR. You should be able to verify this by the ethtool statistic. > >> > >>> Thanks, > >>> -Don Skidmore <donald.c.skidm...@intel.com> > >> > >> > >>>> -----Original Message----- > >>>> From: Łukasz Chrustek [mailto:luk...@chrustek.net] > >>>> Sent: Thursday, October 06, 2016 8:23 AM > >>>> To: Tal Abudi <talab...@gmail.com> > >>>> Cc: e1000-devel@lists.sourceforge.net > >>>> Subject: Re: [E1000-devel] rx queue > >>>> > >>>> Witam, > >>>> > >>>>> Hi > >>>>> What type of system are you running ? > >>>> > >>>> it is ubuntu 16.04 LTS > >>>> > >>>>> Which ethernet controller (lspci -i net) and driver (ethtool -i > >>>>> S8P0) > >>>> # lspci -i net > >>>> 00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon > >>>> E3 v4/Xeon D DMI2 (rev 01) > >>>> 00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 1 (rev 01) > >>>> 00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 2 (rev 01) > >>>> 00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 3 (rev 01) > >>>> 00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 3 (rev 01) > >>>> 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) > >>>> 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) > >>>> 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) > >>>> 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) > >>>> 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) > >>>> 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) > >>>> 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) > >>>> 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) > >>>> 00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01) > >>>> 00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01) > >>>> 00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01) > >>>> 00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D I/O APIC (rev 01) > >>>> 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series > >>>> chipset SPSR (rev 05) > >>>> 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset > >>>> sSATA Controller [AHCI mode] (rev 05) > >>>> 00:14.0 USB controller: Intel Corporation C610/X99 series chipset > >>>> USB xHCI Host Controller (rev 05) > >>>> 00:16.0 Communication controller: Intel Corporation C610/X99 series > >>>> chipset MEI Controller (rev 05) > >>>> 00:16.1 Communication controller: Intel Corporation C610/X99 series > >>>> chipset MEI Controller (rev 05) > >>>> 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset > >>>> USB Enhanced Host Controller (rev 05) > >>>> 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI > >>>> Express Root Port (rev d5) > >>>> 00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI > >>>> Express Root Port (rev d5) > >>>> 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset > >>>> USB Enhanced Host Controller (rev 05) > >>>> 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC > >>>> Controller (rev 05) > >>>> 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset > >>>> 6-Port SATA Controller [AHCI mode] (rev 05) > >>>> 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus > >>>> Controller (rev 05) > >>>> 01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network > >>>> Connection (rev 01) > >>>> 04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network > >>>> Connection (rev 01) > >>>> 06:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI > >>>> Bridge (rev > >>>> 03) > >>>> 07:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED > >>>> Graphics Family (rev 30) > >>>> 7f:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> 7f:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> 7f:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> 7f:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> 7f:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> 7f:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> 7f:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> 7f:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 01) > >>>> 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0d.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0d.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> 7f:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01) > >>>> 7f:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R2PCIe Agent (rev 01) > >>>> 7f:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Ubox (rev 01) > >>>> 7f:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Ubox (rev 01) > >>>> 7f:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Ubox (rev 01) > >>>> 7f:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01) > >>>> 7f:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Home Agent 0 (rev 01) > >>>> 7f:12.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Home Agent 1 (rev 01) > >>>> 7f:12.5 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Home Agent 1 (rev 01) > >>>> 7f:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target > >>>> Address/Thermal/RAS (rev 01) > >>>> 7f:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target > >>>> Address/Thermal/RAS (rev 01) > >>>> 7f:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address > >>>> Decoder (rev 01) > >>>> 7f:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address > >>>> Decoder (rev 01) > >>>> 7f:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01) > >>>> 7f:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01) > >>>> 7f:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal > >>>> Control (rev 01) > >>>> 7f:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal > >>>> Control (rev 01) > >>>> 7f:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01) > >>>> 7f:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01) > >>>> 7f:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> 7f:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> 7f:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> 7f:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> 7f:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01) > >>>> 7f:16.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01) > >>>> 7f:16.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Channel Target Address Decoder (rev 01) > >>>> 7f:16.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Channel Target Address Decoder (rev 01) > >>>> 7f:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01) > >>>> 7f:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01) > >>>> 7f:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal > >>>> Control (rev 01) > >>>> 7f:17.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Thermal > >>>> Control (rev 01) > >>>> 7f:17.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Error (rev 01) > >>>> 7f:17.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Error (rev 01) > >>>> 7f:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> 7f:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> 7f:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> 7f:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> 7f:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 7f:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> 80:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 1 (rev 01) > >>>> 80:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D PCI Express Root Port 3 (rev 01) > >>>> 80:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) > >>>> 80:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) > >>>> 80:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) > >>>> 80:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) > >>>> 80:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) > >>>> 80:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) > >>>> 80:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) > >>>> 80:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) > >>>> 80:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01) > >>>> 80:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01) > >>>> 80:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01) > >>>> 80:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 > >>>> v4/Xeon D I/O APIC (rev 01) > >>>> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> 82:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>> SFI/SFP+ Network Connection (rev 01) > >>>> ff:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> ff:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> ff:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) > >>>> ff:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> ff:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> ff:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) > >>>> ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) > >>>> ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 01) > >>>> ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0d.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0d.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Caching Agent (rev 01) > >>>> ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01) > >>>> ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D R2PCIe Agent (rev 01) > >>>> ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Ubox (rev 01) > >>>> ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Ubox (rev 01) > >>>> ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Ubox (rev 01) > >>>> ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01) > >>>> ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Home Agent 0 (rev 01) > >>>> ff:12.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Home Agent 1 (rev 01) > >>>> ff:12.5 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon > >>>> E3 v4/Xeon D Home Agent 1 (rev 01) > >>>> ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target > >>>> Address/Thermal/RAS (rev 01) > >>>> ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target > >>>> Address/Thermal/RAS (rev 01) > >>>> ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address > >>>> Decoder (rev 01) > >>>> ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address > >>>> Decoder (rev 01) > >>>> ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01) > >>>> ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01) > >>>> ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal > >>>> Control (rev 01) > >>>> ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal > >>>> Control (rev 01) > >>>> ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01) > >>>> ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01) > >>>> ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01) > >>>> ff:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01) > >>>> ff:16.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01) > >>>> ff:16.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Channel Target Address Decoder (rev 01) > >>>> ff:16.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Channel Target Address Decoder (rev 01) > >>>> ff:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01) > >>>> ff:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01) > >>>> ff:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal > >>>> Control (rev 01) > >>>> ff:17.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Thermal > >>>> Control (rev 01) > >>>> ff:17.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Error (rev 01) > >>>> ff:17.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Error (rev 01) > >>>> ff:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> ff:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> ff:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> ff:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01) > >>>> ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 > >>>> v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01) > >>>> > >>>> > >>>> # ethtool -i S8P0 > >>>> driver: ixgbe > >>>> version: 4.4.6 > >>>> firmware-version: 0x61c10001 > >>>> expansion-rom-version: > >>>> bus-info: 0000:82:00.0 > >>>> supports-statistics: yes > >>>> supports-test: yes > >>>> supports-eeprom-access: yes > >>>> supports-register-dump: yes > >>>> supports-priv-flags: yes > >>>> > >>>> Regards > >>>> Lukasz > >>>> > >>>> > >>>>> Thanks, > >>>>> Tal > >>>> > >>>>> On Thu, Oct 6, 2016 at 1:06 PM, Łukasz Chrustek > >>>>> <luk...@chrustek.net> > >>>> wrote: > >>>> > >>>>> Hello, > >>>> > >>>>> I have strange problem, my system has two 12-core procesors, > >>>>> when setting smp_affinity and rps_cpus to spread load accros > >>>>> all cores, I get following: > >>>> > >>>>> ethtool -S S8P0 |grep -E "rx.*pack" > >>>>> rx_packets: 28059349668 > >>>>> rx_fcoe_packets: 0 > >>>>> rx_queue_0_packets: 1739671933 > >>>>> rx_queue_1_packets: 1740406739 > >>>>> rx_queue_2_packets: 1945709824 > >>>>> rx_queue_3_packets: 1741279872 > >>>>> rx_queue_4_packets: 1739593471 > >>>>> rx_queue_5_packets: 1740392727 > >>>>> rx_queue_6_packets: 1740513001 > >>>>> rx_queue_7_packets: 1740921491 > >>>>> rx_queue_8_packets: 1741525459 > >>>>> rx_queue_9_packets: 1739646341 > >>>>> rx_queue_10_packets: 1741068593 > >>>>> rx_queue_11_packets: 1741374592 > >>>>> rx_queue_12_packets: 1739709430 > >>>>> rx_queue_13_packets: 1747586478 > >>>>> rx_queue_14_packets: 1740398506 > >>>>> rx_queue_15_packets: 1739546993 > >>>>> rx_queue_16_packets: 690 > >>>>> rx_queue_17_packets: 455 > >>>>> rx_queue_18_packets: 653 > >>>>> rx_queue_19_packets: 411 > >>>>> rx_queue_20_packets: 510 > >>>>> rx_queue_21_packets: 514 > >>>>> rx_queue_22_packets: 588 > >>>>> rx_queue_23_packets: 431 > >>>> > >>>> > >>>>> but for tx: > >>>> > >>>> > >>>>> ethtool -S S8P0 |grep -E "tx.*pack" > >>>>> tx_packets: 23697264990 > >>>>> tx_fcoe_packets: 0 > >>>>> tx_queue_0_packets: 897068903 > >>>>> tx_queue_1_packets: 897204633 > >>>>> tx_queue_2_packets: 1705630673 > >>>>> tx_queue_3_packets: 897124425 > >>>>> tx_queue_4_packets: 897124753 > >>>>> tx_queue_5_packets: 897114279 > >>>>> tx_queue_6_packets: 897379101 > >>>>> tx_queue_7_packets: 897218157 > >>>>> tx_queue_8_packets: 904811070 > >>>>> tx_queue_9_packets: 897124394 > >>>>> tx_queue_10_packets: 901837638 > >>>>> tx_queue_11_packets: 897419500 > >>>>> tx_queue_12_packets: 1241034979 > >>>>> tx_queue_13_packets: 1247080319 > >>>>> tx_queue_14_packets: 1247499026 > >>>>> tx_queue_15_packets: 1240739566 > >>>>> tx_queue_16_packets: 891720194 > >>>>> tx_queue_17_packets: 891610281 > >>>>> tx_queue_18_packets: 891814848 > >>>>> tx_queue_19_packets: 891770115 > >>>>> tx_queue_20_packets: 891746387 > >>>>> tx_queue_21_packets: 891680822 > >>>>> tx_queue_22_packets: 891775521 > >>>>> tx_queue_23_packets: 891735428 > >>>> > >>>> > >>>>> Can You provide some clue to spread rx_queue for all cores not > >>>>> only first 16 ? > >>>> > > > > ---------------------------------------------------------------------- > > -------- Check out the vibrant tech community on one of the world's > > most engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel® Ethernet, visit > > http://communities.intel.com/community/wired ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
