Could you please re-run the ethregs with the "-s 5:00.1" instead of the "-d 8086:10fb" option. Using the device ID will give you the results for both function 0 and function 1. I just want to verify which function we are getting the results for.
Thanks, Alex On 02/20/2014 06:28 AM, Scott Silverman wrote: > Using the latest 3.19.1 from e1000.sf.net: > > [root@sec54 ~]# ethtool -i eth5 > driver: ixgbe > version: 3.19.1 > firmware-version: 0x18f60001 > bus-info: 0000:05:00.1 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > [root@sec54 ~]# /usr/src/ethregs-1.16.0/ethregs -d 8086:10fb | grep DCA | > more > DCA_RXCTRL[000] 00001200 > DCA_RXCTRL[001] 1a0002a0 > DCA_RXCTRL[002] 190002a0 > DCA_RXCTRL[003] 170002a0 > <snip> > > > > > Thanks, > > Scott Silverman | IT | Simplex Investments | 312-360-2444 > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 > > > On Wed, Feb 19, 2014 at 3:54 PM, Alexander Duyck < > alexander.h.du...@intel.com> wrote: > >> Scott, >> >> The queue 0 issue and the status of the DCA registers points to the use >> of an older driver. The issue was we were initializing the cpu to 0 for >> the q_vectors when we were allocating them. As a result the queues that >> ended up on CPU zero were not setting the tag correctly. That is why we >> moved the intial value to -1 for the q_vector->cpu value in commit >> 245f292d71d3fdd7536c2e4986769d5b9b48fb7f. Based on the behavior it >> sounds like the driver you have is probably something from before 2013, >> if you have a driver with a version number 3.11.X or greater you >> shouldn't have the issue. >> >> Thanks, >> >> Alex >> >> >> On 02/19/2014 12:49 PM, Scott Silverman wrote: >>> Alex, >>> >>> Thanks for the tip about using ethregs to inspect the DCA registers. >>> However, this has led to another question. Specifically, it seems that >>> while DCA is enabled for most of the queues, those that land on core 0 >>> seem to stay disabled for some reason. >>> >>> Here's a snip from one of my machines with 24 cores (and 24 queues). >>> 001-023 all show enabled, but 000 does not: >>> [root@sec54 ethregs-1.16.0]# ./ethregs -d 8086:10fb | grep DCA | more >>> DCA_RXCTRL[000] 00001200 >>> DCA_RXCTRL[001] 1a0002a0 >>> DCA_RXCTRL[002] 190002a0 >>> DCA_RXCTRL[003] 170002a0 >>> DCA_RXCTRL[004] 160002a0 >>> DCA_RXCTRL[005] 150002a0 >>> DCA_RXCTRL[006] 1b0002a0 >>> DCA_RXCTRL[007] 1a0002a0 >>> DCA_RXCTRL[008] 190002a0 >>> DCA_RXCTRL[009] 170002a0 >>> DCA_RXCTRL[010] 160002a0 >>> DCA_RXCTRL[011] 150002a0 >>> DCA_RXCTRL[012] 1b0002a0 >>> DCA_RXCTRL[013] 1a0002a0 >>> DCA_RXCTRL[014] 190002a0 >>> DCA_RXCTRL[015] 170002a0 >>> DCA_RXCTRL[016] 160002a0 >>> DCA_RXCTRL[017] 150002a0 >>> DCA_RXCTRL[018] 1b0002a0 >>> DCA_RXCTRL[019] 1a0002a0 >>> DCA_RXCTRL[020] 190002a0 >>> DCA_RXCTRL[021] 170002a0 >>> DCA_RXCTRL[022] 160002a0 >>> DCA_RXCTRL[023] 150002a0 >>> DCA_RXCTRL[024] 00001200 >>> DCA_RXCTRL[025] 00001200 >>> >>> The TXCTRL shows the same pattern. >>> >>> I'm not sure what to make of this, and I'm also not sure how to get >>> DCA running on queue 0. >>> >>> >>> >>> >>> Thanks, >>> >>> Scott Silverman | IT | Simplex Investments | 312-360-2444 >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> >>> >>> On Thu, Feb 13, 2014 at 4:18 PM, Alexander Duyck >>> <alexander.h.du...@intel.com <mailto:alexander.h.du...@intel.com>> >> wrote: >>> Odds are the problem is the order the modules are loading in. In >>> order >>> to print the message ioatdma and dca needs to be loaded before >>> ixgbe, if >>> the ioatdma driver is loaded after then you won't see the message >>> as it >>> is only printed at probe time and not when a DCA provider is >>> registered. >>> >>> If you are loading dca, ixgbe, and then ioatdma in that order the >>> ixgbe >>> won't show as having DCA enabled because the ixgbe_notify_dca call >>> doesn't appear to print any message. As a result you would have DCA >>> enabled, but it won't have displayed any message stating as such. >>> >>> The tell-tale sign that you have DCA enabled is to use the ethregs >>> tool >>> to dump the registers for the device. The DCA_RXCTRL and DCA_TXCTRL >>> registers will have a APIC tag ID in the upper 8 bits. Usually the >>> value is something like 1f or 1e. If the value is 00 then it is >>> disabled. Below is a snippet of a register dump on a system I >>> have here >>> that supports DCA. >>> >>> DCA Enabled: >>> DCA_TXCTRL[0] 1f002220 >>> DCA_TXCTRL[1] 1f002220 >>> >>> DCA Disabled: >>> DCA_TXCTRL[0] 00002220 >>> DCA_TXCTRL[1] 00002220 >>> >>> Thanks, >>> >>> Alex >>> >>> On 02/13/2014 01:38 PM, Scott Silverman wrote: >>> > It seems I was making a mistake, just not the one I thought I was. >>> > When I looked at 3.18.7, it was after a system boot. When I >>> looked at >>> > 3.19.1 it was only after removing and reloading the module. >>> > >>> > I've attached dmesg output from a system boot just now, showing >>> dca / >>> > igb / ixgbe / ioatdma modules all loading. It seems that for >>> whatever >>> > reason, during the system boot DCA is not enabled for ixgbe (but >>> it is >>> > for igb). If I remove and reload ixgbe, it then enables DCA. >>> (seen at >>> > the end of the attached dmesg). >>> > >>> > Is this the expected behavior? Am I doing something wrong? Do I >> need >>> > to ensure that ioatdma loads before ixgbe? (if so, why doesn't igb >>> > seem to care?) >>> > >>> > >>> > >>> > >>> > Thanks, >>> > >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444 >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> > >>> > >>> > On Thu, Feb 13, 2014 at 3:18 PM, Scott Silverman >>> > <ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com> >>> > <mailto:ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com>>> wrote: >>> > >>> > Alex/John, >>> > >>> > Thanks for the clarification with regards to DDIO/DCA. >>> > >>> > As far as my results with the 3.18.7 driver, I can't duplicate >>> > them now, so I'll chalk it up to a mistake on my side. Sorry >> for >>> > the trouble. >>> > >>> > >>> > >>> > >>> > Thanks, >>> > >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444 >>> <tel:312-360-2444> >>> > <tel:312-360-2444 <tel:312-360-2444>> >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> > >>> > >>> > On Thu, Feb 13, 2014 at 3:03 PM, Duyck, Alexander H >>> > <alexander.h.du...@intel.com >>> <mailto:alexander.h.du...@intel.com> >>> <mailto:alexander.h.du...@intel.com >>> <mailto:alexander.h.du...@intel.com>>> >>> > wrote: >>> > >>> > Scott, >>> > >>> > >>> > >>> > DDIO does not completely replace DCA. DCA provides >>> > functionality for remote socket, while DDIO only functions >>> > with the local socket for the device. >>> > >>> > >>> > >>> > The ixgbe driver prints that DCA message if DCA was >> detected >>> > when the driver was loaded. Did you try running lsmod >>> to see >>> > if ioatdma and dca modules were loaded when you loaded the >>> > 3.18.7 driver? >>> > >>> > >>> > >>> > Thanks, >>> > >>> > >>> > >>> > Alex >>> > >>> > >>> > >>> > *From:*Scott Silverman >>> > [mailto:ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com> >>> > <mailto:ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com>>] >>> > *Sent:* Thursday, February 13, 2014 12:36 PM >>> > *To:* Duyck, Alexander H >>> > *Cc:* Ronciak, John; e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net> >>> > <mailto:e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net>> >>> > *Subject:* Re: [E1000-devel] DCA on Sandy Bridge? >>> > >>> > >>> > >>> > John, >>> > >>> > >>> > >>> > I think you misunderstood. Sandy Bridge ioatdma/dca >>> should be >>> > supported in kernel 3.4.41. However, I only see that "DCA" >>> > flag in ixgbe/dmesg when I am using ixgbe 3.19.1, not when >> I >>> > am using 3.18.7. I am asking why that might be. Nothing >> else >>> > changed, aside from rmmod ixgbe (3.18.7) and modprobe ixgbe >>> > (3.19.1). >>> > >>> > >>> > >>> > Alex, >>> > >>> > >>> > >>> > There is nothing in dmesg to explain it. In fact, the igb >>> > driver always uses DCA (prints "DCA enabled"), so I know >>> that >>> > from a platform perspective it is working. >>> > >>> > >>> > >>> > To put it simply, I have two questions: >>> > >>> > >>> > >>> > 1. On Sandy bridge (and Ivy Bridge, etc), does DDIO replace >>> > DCA? If so, what implication does the presence of "DCA" in >>> > enabled features on a platform that is meant to have DDIO >>> > instead of DCA? >>> > >>> > 2. What change from 3.18.7 to 3.19.1 would cause the >> feature >>> > to become Enabled on a platform that otherwise supports >> DCA? >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > Thanks, >>> > >>> > >>> > >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444 >>> > <tel:312-360-2444 <tel:312-360-2444>> >>> > >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> > >>> > >>> > >>> > On Thu, Feb 13, 2014 at 1:59 PM, Alexander Duyck >>> > <alexander.h.du...@intel.com >>> <mailto:alexander.h.du...@intel.com> >>> > <mailto:alexander.h.du...@intel.com >>> <mailto:alexander.h.du...@intel.com>>> wrote: >>> > >>> > Scott, >>> > >>> > You might try checking the dmesg log on the systems that >>> come >>> > up as not >>> > supporting DCA. There was a patch submitted a year or >>> so ago >>> > that would >>> > disable DCA on platforms that had a misconfigured APIC >>> ID tag map. >>> > >>> > It is possible that the platform might have had DCA >> disabled >>> > due to this >>> > in the case of the 3.18.7. You might want to go back and >>> > recheck the >>> > dmesg log to verify if DCA was disabled due to a >>> configuration >>> > error in >>> > the BIOS. >>> > >>> > Thanks, >>> > >>> > Alex >>> > >>> > >>> > On 02/13/2014 08:27 AM, Scott Silverman wrote: >>> > > John, >>> > > >>> > > Thanks for the prompt response, but I still have some >>> > questions/concerns. >>> > > >>> > > -My motherboard (SuperMicro X9DRW) specifically provides >> a >>> > firmware option >>> > > to enable "DCA" but you state that the chipset doesn't >>> > include it at all? >>> > > >>> > > -Why is it reported only sometimes? >>> > > >>> > > Proc, Kern, ixgbe - DCA? >>> > > Sandy, 3.4, 3.18.7 - No DCA >>> > > Sandy, 3.4, 3.19.1 - DCA >>> > > Sandy, 3.12, 3.19.1 - DCA >>> > > Ivy, 3.12, 3.18.7 - DCA >>> > > Ivy, 3.12, 3.19.1 - DCA >>> > > >>> > > >>> > > >>> > > >>> > > Thanks, >>> > > >>> > > Scott Silverman | IT | Simplex Investments | 312-360-2444 >>> > <tel:312-360-2444 <tel:312-360-2444>> >>> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> > > >>> > > >>> > > On Mon, Feb 10, 2014 at 4:26 PM, Ronciak, John >>> > <john.ronc...@intel.com <mailto:john.ronc...@intel.com> >>> <mailto:john.ronc...@intel.com <mailto:john.ronc...@intel.com >>>>> wrote: >>> > > >>> > >> The driver is just telling you what's possible not >>> that DCA >>> > is enabled. >>> > >> The newer chipsets do not included it as it's all DDIO. >>> > So even though >>> > >> the NIC's is capable of support DCA the chipset does not >>> > have it so it >>> > >> won't be used. That's all this is saying. It's not an >>> > issue at all. >>> > >> >>> > >> Cheers, >>> > >> John >>> > >> >>> > >> >>> > >>> -----Original Message----- >>> > >>> From: Scott Silverman >>> > [mailto:ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com> >>> > <mailto:ssilver...@simplexinvestments.com >>> <mailto:ssilver...@simplexinvestments.com>>] >>> > >>> Sent: Monday, February 10, 2014 11:59 AM >>> > >>> To: e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net> >>> > <mailto:e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net>> >>> > >>> Subject: [E1000-devel] DCA on Sandy Bridge? >>> > >>> >>> > >>> My understanding is the DDIO completely replaces DCA on >>> > sandy bridge >>> > >>> and newer hardware (E5+ xeons). I expected this is why >> I >>> > don't see >>> > >>> "DCA" listed when I load ixgbe, as I would on my >> nehalem >>> > systems. >>> > >>> >>> > >>> I see the following when loading ixgbe 3.18.7 on an >>> > E5-2670 system >>> > >>> (kernel >>> > >>> 3.4.41): >>> > >>> >>> > >>> ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) >>> > 00:1b:21:5c:66:0d >>> > >>> ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, >>> PBA No: >>> > E68793-002 >>> > >>> ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 32 >> TxQ: >>> > 32 FdirHash >>> > >>> RSC ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit >>> Network >>> > Connection >>> > >>> >>> > >>> If I take the same system, unload ixgbe, and load ixgbe >>> > 3.19.1, I see >>> > >>> this (emphasis mine): >>> > >>> >>> > >>> ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s >>> > available ixgbe >>> > >>> 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding >>> > Loss:20%) ixgbe >>> > >>> 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No: >>> > E68793-002 ixgbe >>> > >>> 0000:04:00.1: 00:1b:21:5c:66:0d ixgbe 0000:04:00.1: >>> eth3: >>> > Enabled >>> > >>> Features: RxQ: 32 TxQ: 32 FdirHash *DCA*RSC ixgbe >>> > 0000:04:00.1: eth3: >>> > >>> Intel(R) 10 Gigabit Network Connection >>> > >>> >>> > >>> Can anyone explain why I see "DCA" with 3.19.1 and not >>> > with 3.18.7. >>> > >>> Also, does it matter? >>> > >>> >>> > >>> Thanks, >>> > >>> >>> > >>> Scott Silverman | IT | Simplex Investments | >>> 312-360-2444 >>> > <tel:312-360-2444 <tel:312-360-2444>> >>> > >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604 >>> > >> >>> > > >>> > >>> > >>> > >>> > >>> > >>> >>> >> ------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
