Scott, The queue 0 issue and the status of the DCA registers points to the use of an older driver. The issue was we were initializing the cpu to 0 for the q_vectors when we were allocating them. As a result the queues that ended up on CPU zero were not setting the tag correctly. That is why we moved the intial value to -1 for the q_vector->cpu value in commit 245f292d71d3fdd7536c2e4986769d5b9b48fb7f. Based on the behavior it sounds like the driver you have is probably something from before 2013, if you have a driver with a version number 3.11.X or greater you shouldn't have the issue.
Thanks, Alex On 02/19/2014 12:49 PM, Scott Silverman wrote: > Alex, > > Thanks for the tip about using ethregs to inspect the DCA registers. > However, this has led to another question. Specifically, it seems that > while DCA is enabled for most of the queues, those that land on core 0 > seem to stay disabled for some reason. > > Here's a snip from one of my machines with 24 cores (and 24 queues). > 001-023 all show enabled, but 000 does not: > [root@sec54 ethregs-1.16.0]# ./ethregs -d 8086:10fb | grep DCA | more > DCA_RXCTRL[000] 00001200 > DCA_RXCTRL[001] 1a0002a0 > DCA_RXCTRL[002] 190002a0 > DCA_RXCTRL[003] 170002a0 > DCA_RXCTRL[004] 160002a0 > DCA_RXCTRL[005] 150002a0 > DCA_RXCTRL[006] 1b0002a0 > DCA_RXCTRL[007] 1a0002a0 > DCA_RXCTRL[008] 190002a0 > DCA_RXCTRL[009] 170002a0 > DCA_RXCTRL[010] 160002a0 > DCA_RXCTRL[011] 150002a0 > DCA_RXCTRL[012] 1b0002a0 > DCA_RXCTRL[013] 1a0002a0 > DCA_RXCTRL[014] 190002a0 > DCA_RXCTRL[015] 170002a0 > DCA_RXCTRL[016] 160002a0 > DCA_RXCTRL[017] 150002a0 > DCA_RXCTRL[018] 1b0002a0 > DCA_RXCTRL[019] 1a0002a0 > DCA_RXCTRL[020] 190002a0 > DCA_RXCTRL[021] 170002a0 > DCA_RXCTRL[022] 160002a0 > DCA_RXCTRL[023] 150002a0 > DCA_RXCTRL[024] 00001200 > DCA_RXCTRL[025] 00001200 > > The TXCTRL shows the same pattern. > > I'm not sure what to make of this, and I'm also not sure how to get > DCA running on queue 0. > > > > > Thanks, > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > On Thu, Feb 13, 2014 at 4:18 PM, Alexander Duyck > <alexander.h.du...@intel.com <mailto:alexander.h.du...@intel.com>> wrote: > > Odds are the problem is the order the modules are loading in. In > order > to print the message ioatdma and dca needs to be loaded before > ixgbe, if > the ioatdma driver is loaded after then you won't see the message > as it > is only printed at probe time and not when a DCA provider is > registered. > > If you are loading dca, ixgbe, and then ioatdma in that order the > ixgbe > won't show as having DCA enabled because the ixgbe_notify_dca call > doesn't appear to print any message. As a result you would have DCA > enabled, but it won't have displayed any message stating as such. > > The tell-tale sign that you have DCA enabled is to use the ethregs > tool > to dump the registers for the device. The DCA_RXCTRL and DCA_TXCTRL > registers will have a APIC tag ID in the upper 8 bits. Usually the > value is something like 1f or 1e. If the value is 00 then it is > disabled. Below is a snippet of a register dump on a system I > have here > that supports DCA. > > DCA Enabled: > DCA_TXCTRL[0] 1f002220 > DCA_TXCTRL[1] 1f002220 > > DCA Disabled: > DCA_TXCTRL[0] 00002220 > DCA_TXCTRL[1] 00002220 > > Thanks, > > Alex > > On 02/13/2014 01:38 PM, Scott Silverman wrote: > > It seems I was making a mistake, just not the one I thought I was. > > When I looked at 3.18.7, it was after a system boot. When I > looked at > > 3.19.1 it was only after removing and reloading the module. > > > > I've attached dmesg output from a system boot just now, showing > dca / > > igb / ixgbe / ioatdma modules all loading. It seems that for > whatever > > reason, during the system boot DCA is not enabled for ixgbe (but > it is > > for igb). If I remove and reload ixgbe, it then enables DCA. > (seen at > > the end of the attached dmesg). > > > > Is this the expected behavior? Am I doing something wrong? Do I need > > to ensure that ioatdma loads before ixgbe? (if so, why doesn't igb > > seem to care?) > > > > > > > > > > Thanks, > > > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > > > > On Thu, Feb 13, 2014 at 3:18 PM, Scott Silverman > > <ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com> > > <mailto:ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com>>> wrote: > > > > Alex/John, > > > > Thanks for the clarification with regards to DDIO/DCA. > > > > As far as my results with the 3.18.7 driver, I can't duplicate > > them now, so I'll chalk it up to a mistake on my side. Sorry for > > the trouble. > > > > > > > > > > Thanks, > > > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > <tel:312-360-2444> > > <tel:312-360-2444 <tel:312-360-2444>> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > > > > On Thu, Feb 13, 2014 at 3:03 PM, Duyck, Alexander H > > <alexander.h.du...@intel.com > <mailto:alexander.h.du...@intel.com> > <mailto:alexander.h.du...@intel.com > <mailto:alexander.h.du...@intel.com>>> > > wrote: > > > > Scott, > > > > > > > > DDIO does not completely replace DCA. DCA provides > > functionality for remote socket, while DDIO only functions > > with the local socket for the device. > > > > > > > > The ixgbe driver prints that DCA message if DCA was detected > > when the driver was loaded. Did you try running lsmod > to see > > if ioatdma and dca modules were loaded when you loaded the > > 3.18.7 driver? > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > *From:*Scott Silverman > > [mailto:ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com> > > <mailto:ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com>>] > > *Sent:* Thursday, February 13, 2014 12:36 PM > > *To:* Duyck, Alexander H > > *Cc:* Ronciak, John; e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > <mailto:e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net>> > > *Subject:* Re: [E1000-devel] DCA on Sandy Bridge? > > > > > > > > John, > > > > > > > > I think you misunderstood. Sandy Bridge ioatdma/dca > should be > > supported in kernel 3.4.41. However, I only see that "DCA" > > flag in ixgbe/dmesg when I am using ixgbe 3.19.1, not when I > > am using 3.18.7. I am asking why that might be. Nothing else > > changed, aside from rmmod ixgbe (3.18.7) and modprobe ixgbe > > (3.19.1). > > > > > > > > Alex, > > > > > > > > There is nothing in dmesg to explain it. In fact, the igb > > driver always uses DCA (prints "DCA enabled"), so I know > that > > from a platform perspective it is working. > > > > > > > > To put it simply, I have two questions: > > > > > > > > 1. On Sandy bridge (and Ivy Bridge, etc), does DDIO replace > > DCA? If so, what implication does the presence of "DCA" in > > enabled features on a platform that is meant to have DDIO > > instead of DCA? > > > > 2. What change from 3.18.7 to 3.19.1 would cause the feature > > to become Enabled on a platform that otherwise supports DCA? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > > <tel:312-360-2444 <tel:312-360-2444>> > > > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > > > > > > On Thu, Feb 13, 2014 at 1:59 PM, Alexander Duyck > > <alexander.h.du...@intel.com > <mailto:alexander.h.du...@intel.com> > > <mailto:alexander.h.du...@intel.com > <mailto:alexander.h.du...@intel.com>>> wrote: > > > > Scott, > > > > You might try checking the dmesg log on the systems that > come > > up as not > > supporting DCA. There was a patch submitted a year or > so ago > > that would > > disable DCA on platforms that had a misconfigured APIC > ID tag map. > > > > It is possible that the platform might have had DCA disabled > > due to this > > in the case of the 3.18.7. You might want to go back and > > recheck the > > dmesg log to verify if DCA was disabled due to a > configuration > > error in > > the BIOS. > > > > Thanks, > > > > Alex > > > > > > On 02/13/2014 08:27 AM, Scott Silverman wrote: > > > John, > > > > > > Thanks for the prompt response, but I still have some > > questions/concerns. > > > > > > -My motherboard (SuperMicro X9DRW) specifically provides a > > firmware option > > > to enable "DCA" but you state that the chipset doesn't > > include it at all? > > > > > > -Why is it reported only sometimes? > > > > > > Proc, Kern, ixgbe - DCA? > > > Sandy, 3.4, 3.18.7 - No DCA > > > Sandy, 3.4, 3.19.1 - DCA > > > Sandy, 3.12, 3.19.1 - DCA > > > Ivy, 3.12, 3.18.7 - DCA > > > Ivy, 3.12, 3.19.1 - DCA > > > > > > > > > > > > > > > Thanks, > > > > > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > > <tel:312-360-2444 <tel:312-360-2444>> > > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > > > > > > > On Mon, Feb 10, 2014 at 4:26 PM, Ronciak, John > > <john.ronc...@intel.com <mailto:john.ronc...@intel.com> > <mailto:john.ronc...@intel.com <mailto:john.ronc...@intel.com>>>wrote: > > > > > >> The driver is just telling you what's possible not > that DCA > > is enabled. > > >> The newer chipsets do not included it as it's all DDIO. > > So even though > > >> the NIC's is capable of support DCA the chipset does not > > have it so it > > >> won't be used. That's all this is saying. It's not an > > issue at all. > > >> > > >> Cheers, > > >> John > > >> > > >> > > >>> -----Original Message----- > > >>> From: Scott Silverman > > [mailto:ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com> > > <mailto:ssilver...@simplexinvestments.com > <mailto:ssilver...@simplexinvestments.com>>] > > >>> Sent: Monday, February 10, 2014 11:59 AM > > >>> To: e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > <mailto:e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net>> > > >>> Subject: [E1000-devel] DCA on Sandy Bridge? > > >>> > > >>> My understanding is the DDIO completely replaces DCA on > > sandy bridge > > >>> and newer hardware (E5+ xeons). I expected this is why I > > don't see > > >>> "DCA" listed when I load ixgbe, as I would on my nehalem > > systems. > > >>> > > >>> I see the following when loading ixgbe 3.18.7 on an > > E5-2670 system > > >>> (kernel > > >>> 3.4.41): > > >>> > > >>> ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) > > 00:1b:21:5c:66:0d > > >>> ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, > PBA No: > > E68793-002 > > >>> ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 32 TxQ: > > 32 FdirHash > > >>> RSC ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit > Network > > Connection > > >>> > > >>> If I take the same system, unload ixgbe, and load ixgbe > > 3.19.1, I see > > >>> this (emphasis mine): > > >>> > > >>> ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s > > available ixgbe > > >>> 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding > > Loss:20%) ixgbe > > >>> 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No: > > E68793-002 ixgbe > > >>> 0000:04:00.1: 00:1b:21:5c:66:0d ixgbe 0000:04:00.1: > eth3: > > Enabled > > >>> Features: RxQ: 32 TxQ: 32 FdirHash *DCA*RSC ixgbe > > 0000:04:00.1: eth3: > > >>> Intel(R) 10 Gigabit Network Connection > > >>> > > >>> Can anyone explain why I see "DCA" with 3.19.1 and not > > with 3.18.7. > > >>> Also, does it matter? > > >>> > > >>> Thanks, > > >>> > > >>> Scott Silverman | IT | Simplex Investments | > 312-360-2444 > > <tel:312-360-2444 <tel:312-360-2444>> > > >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > >> > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
