Using the latest 3.19.1 from e1000.sf.net:
[root@sec54 ~]# ethtool -i eth5
driver: ixgbe
version: 3.19.1
firmware-version: 0x18f60001
bus-info: 0000:05:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[root@sec54 ~]# /usr/src/ethregs-1.16.0/ethregs -d 8086:10fb | grep DCA |
more
DCA_RXCTRL[000] 00001200
DCA_RXCTRL[001] 1a0002a0
DCA_RXCTRL[002] 190002a0
DCA_RXCTRL[003] 170002a0
<snip>
Thanks,
Scott Silverman | IT | Simplex Investments | 312-360-2444
230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
On Wed, Feb 19, 2014 at 3:54 PM, Alexander Duyck <
alexander.h.du...@intel.com> wrote:
> Scott,
>
> The queue 0 issue and the status of the DCA registers points to the use
> of an older driver. The issue was we were initializing the cpu to 0 for
> the q_vectors when we were allocating them. As a result the queues that
> ended up on CPU zero were not setting the tag correctly. That is why we
> moved the intial value to -1 for the q_vector->cpu value in commit
> 245f292d71d3fdd7536c2e4986769d5b9b48fb7f. Based on the behavior it
> sounds like the driver you have is probably something from before 2013,
> if you have a driver with a version number 3.11.X or greater you
> shouldn't have the issue.
>
> Thanks,
>
> Alex
>
>
> On 02/19/2014 12:49 PM, Scott Silverman wrote:
> > Alex,
> >
> > Thanks for the tip about using ethregs to inspect the DCA registers.
> > However, this has led to another question. Specifically, it seems that
> > while DCA is enabled for most of the queues, those that land on core 0
> > seem to stay disabled for some reason.
> >
> > Here's a snip from one of my machines with 24 cores (and 24 queues).
> > 001-023 all show enabled, but 000 does not:
> > [root@sec54 ethregs-1.16.0]# ./ethregs -d 8086:10fb | grep DCA | more
> > DCA_RXCTRL[000] 00001200
> > DCA_RXCTRL[001] 1a0002a0
> > DCA_RXCTRL[002] 190002a0
> > DCA_RXCTRL[003] 170002a0
> > DCA_RXCTRL[004] 160002a0
> > DCA_RXCTRL[005] 150002a0
> > DCA_RXCTRL[006] 1b0002a0
> > DCA_RXCTRL[007] 1a0002a0
> > DCA_RXCTRL[008] 190002a0
> > DCA_RXCTRL[009] 170002a0
> > DCA_RXCTRL[010] 160002a0
> > DCA_RXCTRL[011] 150002a0
> > DCA_RXCTRL[012] 1b0002a0
> > DCA_RXCTRL[013] 1a0002a0
> > DCA_RXCTRL[014] 190002a0
> > DCA_RXCTRL[015] 170002a0
> > DCA_RXCTRL[016] 160002a0
> > DCA_RXCTRL[017] 150002a0
> > DCA_RXCTRL[018] 1b0002a0
> > DCA_RXCTRL[019] 1a0002a0
> > DCA_RXCTRL[020] 190002a0
> > DCA_RXCTRL[021] 170002a0
> > DCA_RXCTRL[022] 160002a0
> > DCA_RXCTRL[023] 150002a0
> > DCA_RXCTRL[024] 00001200
> > DCA_RXCTRL[025] 00001200
> >
> > The TXCTRL shows the same pattern.
> >
> > I'm not sure what to make of this, and I'm also not sure how to get
> > DCA running on queue 0.
> >
> >
> >
> >
> > Thanks,
> >
> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >
> >
> > On Thu, Feb 13, 2014 at 4:18 PM, Alexander Duyck
> > <alexander.h.du...@intel.com <mailto:alexander.h.du...@intel.com>>
> wrote:
> >
> > Odds are the problem is the order the modules are loading in. In
> > order
> > to print the message ioatdma and dca needs to be loaded before
> > ixgbe, if
> > the ioatdma driver is loaded after then you won't see the message
> > as it
> > is only printed at probe time and not when a DCA provider is
> > registered.
> >
> > If you are loading dca, ixgbe, and then ioatdma in that order the
> > ixgbe
> > won't show as having DCA enabled because the ixgbe_notify_dca call
> > doesn't appear to print any message. As a result you would have DCA
> > enabled, but it won't have displayed any message stating as such.
> >
> > The tell-tale sign that you have DCA enabled is to use the ethregs
> > tool
> > to dump the registers for the device. The DCA_RXCTRL and DCA_TXCTRL
> > registers will have a APIC tag ID in the upper 8 bits. Usually the
> > value is something like 1f or 1e. If the value is 00 then it is
> > disabled. Below is a snippet of a register dump on a system I
> > have here
> > that supports DCA.
> >
> > DCA Enabled:
> > DCA_TXCTRL[0] 1f002220
> > DCA_TXCTRL[1] 1f002220
> >
> > DCA Disabled:
> > DCA_TXCTRL[0] 00002220
> > DCA_TXCTRL[1] 00002220
> >
> > Thanks,
> >
> > Alex
> >
> > On 02/13/2014 01:38 PM, Scott Silverman wrote:
> > > It seems I was making a mistake, just not the one I thought I was.
> > > When I looked at 3.18.7, it was after a system boot. When I
> > looked at
> > > 3.19.1 it was only after removing and reloading the module.
> > >
> > > I've attached dmesg output from a system boot just now, showing
> > dca /
> > > igb / ixgbe / ioatdma modules all loading. It seems that for
> > whatever
> > > reason, during the system boot DCA is not enabled for ixgbe (but
> > it is
> > > for igb). If I remove and reload ixgbe, it then enables DCA.
> > (seen at
> > > the end of the attached dmesg).
> > >
> > > Is this the expected behavior? Am I doing something wrong? Do I
> need
> > > to ensure that ioatdma loads before ixgbe? (if so, why doesn't igb
> > > seem to care?)
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > >
> > >
> > > On Thu, Feb 13, 2014 at 3:18 PM, Scott Silverman
> > > <ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>
> > > <mailto:ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>>> wrote:
> > >
> > > Alex/John,
> > >
> > > Thanks for the clarification with regards to DDIO/DCA.
> > >
> > > As far as my results with the 3.18.7 driver, I can't duplicate
> > > them now, so I'll chalk it up to a mistake on my side. Sorry
> for
> > > the trouble.
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > <tel:312-360-2444>
> > > <tel:312-360-2444 <tel:312-360-2444>>
> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > >
> > >
> > > On Thu, Feb 13, 2014 at 3:03 PM, Duyck, Alexander H
> > > <alexander.h.du...@intel.com
> > <mailto:alexander.h.du...@intel.com>
> > <mailto:alexander.h.du...@intel.com
> > <mailto:alexander.h.du...@intel.com>>>
> > > wrote:
> > >
> > > Scott,
> > >
> > >
> > >
> > > DDIO does not completely replace DCA. DCA provides
> > > functionality for remote socket, while DDIO only functions
> > > with the local socket for the device.
> > >
> > >
> > >
> > > The ixgbe driver prints that DCA message if DCA was
> detected
> > > when the driver was loaded. Did you try running lsmod
> > to see
> > > if ioatdma and dca modules were loaded when you loaded the
> > > 3.18.7 driver?
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > > *From:*Scott Silverman
> > > [mailto:ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>
> > > <mailto:ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>>]
> > > *Sent:* Thursday, February 13, 2014 12:36 PM
> > > *To:* Duyck, Alexander H
> > > *Cc:* Ronciak, John; e1000-devel@lists.sourceforge.net
> > <mailto:e1000-devel@lists.sourceforge.net>
> > > <mailto:e1000-devel@lists.sourceforge.net
> > <mailto:e1000-devel@lists.sourceforge.net>>
> > > *Subject:* Re: [E1000-devel] DCA on Sandy Bridge?
> > >
> > >
> > >
> > > John,
> > >
> > >
> > >
> > > I think you misunderstood. Sandy Bridge ioatdma/dca
> > should be
> > > supported in kernel 3.4.41. However, I only see that "DCA"
> > > flag in ixgbe/dmesg when I am using ixgbe 3.19.1, not when
> I
> > > am using 3.18.7. I am asking why that might be. Nothing
> else
> > > changed, aside from rmmod ixgbe (3.18.7) and modprobe ixgbe
> > > (3.19.1).
> > >
> > >
> > >
> > > Alex,
> > >
> > >
> > >
> > > There is nothing in dmesg to explain it. In fact, the igb
> > > driver always uses DCA (prints "DCA enabled"), so I know
> > that
> > > from a platform perspective it is working.
> > >
> > >
> > >
> > > To put it simply, I have two questions:
> > >
> > >
> > >
> > > 1. On Sandy bridge (and Ivy Bridge, etc), does DDIO replace
> > > DCA? If so, what implication does the presence of "DCA" in
> > > enabled features on a platform that is meant to have DDIO
> > > instead of DCA?
> > >
> > > 2. What change from 3.18.7 to 3.19.1 would cause the
> feature
> > > to become Enabled on a platform that otherwise supports
> DCA?
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > > <tel:312-360-2444 <tel:312-360-2444>>
> > >
> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > >
> > >
> > >
> > > On Thu, Feb 13, 2014 at 1:59 PM, Alexander Duyck
> > > <alexander.h.du...@intel.com
> > <mailto:alexander.h.du...@intel.com>
> > > <mailto:alexander.h.du...@intel.com
> > <mailto:alexander.h.du...@intel.com>>> wrote:
> > >
> > > Scott,
> > >
> > > You might try checking the dmesg log on the systems that
> > come
> > > up as not
> > > supporting DCA. There was a patch submitted a year or
> > so ago
> > > that would
> > > disable DCA on platforms that had a misconfigured APIC
> > ID tag map.
> > >
> > > It is possible that the platform might have had DCA
> disabled
> > > due to this
> > > in the case of the 3.18.7. You might want to go back and
> > > recheck the
> > > dmesg log to verify if DCA was disabled due to a
> > configuration
> > > error in
> > > the BIOS.
> > >
> > > Thanks,
> > >
> > > Alex
> > >
> > >
> > > On 02/13/2014 08:27 AM, Scott Silverman wrote:
> > > > John,
> > > >
> > > > Thanks for the prompt response, but I still have some
> > > questions/concerns.
> > > >
> > > > -My motherboard (SuperMicro X9DRW) specifically provides
> a
> > > firmware option
> > > > to enable "DCA" but you state that the chipset doesn't
> > > include it at all?
> > > >
> > > > -Why is it reported only sometimes?
> > > >
> > > > Proc, Kern, ixgbe - DCA?
> > > > Sandy, 3.4, 3.18.7 - No DCA
> > > > Sandy, 3.4, 3.19.1 - DCA
> > > > Sandy, 3.12, 3.19.1 - DCA
> > > > Ivy, 3.12, 3.18.7 - DCA
> > > > Ivy, 3.12, 3.19.1 - DCA
> > > >
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > > <tel:312-360-2444 <tel:312-360-2444>>
> > > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > > >
> > > >
> > > > On Mon, Feb 10, 2014 at 4:26 PM, Ronciak, John
> > > <john.ronc...@intel.com <mailto:john.ronc...@intel.com>
> > <mailto:john.ronc...@intel.com <mailto:john.ronc...@intel.com
> >>>wrote:
> > > >
> > > >> The driver is just telling you what's possible not
> > that DCA
> > > is enabled.
> > > >> The newer chipsets do not included it as it's all DDIO.
> > > So even though
> > > >> the NIC's is capable of support DCA the chipset does not
> > > have it so it
> > > >> won't be used. That's all this is saying. It's not an
> > > issue at all.
> > > >>
> > > >> Cheers,
> > > >> John
> > > >>
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: Scott Silverman
> > > [mailto:ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>
> > > <mailto:ssilver...@simplexinvestments.com
> > <mailto:ssilver...@simplexinvestments.com>>]
> > > >>> Sent: Monday, February 10, 2014 11:59 AM
> > > >>> To: e1000-devel@lists.sourceforge.net
> > <mailto:e1000-devel@lists.sourceforge.net>
> > > <mailto:e1000-devel@lists.sourceforge.net
> > <mailto:e1000-devel@lists.sourceforge.net>>
> > > >>> Subject: [E1000-devel] DCA on Sandy Bridge?
> > > >>>
> > > >>> My understanding is the DDIO completely replaces DCA on
> > > sandy bridge
> > > >>> and newer hardware (E5+ xeons). I expected this is why
> I
> > > don't see
> > > >>> "DCA" listed when I load ixgbe, as I would on my
> nehalem
> > > systems.
> > > >>>
> > > >>> I see the following when loading ixgbe 3.18.7 on an
> > > E5-2670 system
> > > >>> (kernel
> > > >>> 3.4.41):
> > > >>>
> > > >>> ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8)
> > > 00:1b:21:5c:66:0d
> > > >>> ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4,
> > PBA No:
> > > E68793-002
> > > >>> ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 32
> TxQ:
> > > 32 FdirHash
> > > >>> RSC ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit
> > Network
> > > Connection
> > > >>>
> > > >>> If I take the same system, unload ixgbe, and load ixgbe
> > > 3.19.1, I see
> > > >>> this (emphasis mine):
> > > >>>
> > > >>> ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s
> > > available ixgbe
> > > >>> 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding
> > > Loss:20%) ixgbe
> > > >>> 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No:
> > > E68793-002 ixgbe
> > > >>> 0000:04:00.1: 00:1b:21:5c:66:0d ixgbe 0000:04:00.1:
> > eth3:
> > > Enabled
> > > >>> Features: RxQ: 32 TxQ: 32 FdirHash *DCA*RSC ixgbe
> > > 0000:04:00.1: eth3:
> > > >>> Intel(R) 10 Gigabit Network Connection
> > > >>>
> > > >>> Can anyone explain why I see "DCA" with 3.19.1 and not
> > > with 3.18.7.
> > > >>> Also, does it matter?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Scott Silverman | IT | Simplex Investments |
> > 312-360-2444
> > > <tel:312-360-2444 <tel:312-360-2444>>
> > > >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
>
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
