Sure, I've attached the output.
Thanks,
Scott Silverman | IT | Simplex Investments | 312-360-2444
230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
On Thu, Feb 20, 2014 at 11:10 AM, Alexander Duyck <
alexander.h.du...@intel.com> wrote:
> Could you please re-run the ethregs with the "-s 5:00.1" instead of the
> "-d 8086:10fb" option. Using the device ID will give you the results
> for both function 0 and function 1. I just want to verify which
> function we are getting the results for.
>
> Thanks,
>
> Alex
>
> On 02/20/2014 06:28 AM, Scott Silverman wrote:
> > Using the latest 3.19.1 from e1000.sf.net:
> >
> > [root@sec54 ~]# ethtool -i eth5
> > driver: ixgbe
> > version: 3.19.1
> > firmware-version: 0x18f60001
> > bus-info: 0000:05:00.1
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> > [root@sec54 ~]# /usr/src/ethregs-1.16.0/ethregs -d 8086:10fb | grep DCA
> |
> > more
> > DCA_RXCTRL[000] 00001200
> > DCA_RXCTRL[001] 1a0002a0
> > DCA_RXCTRL[002] 190002a0
> > DCA_RXCTRL[003] 170002a0
> > <snip>
> >
> >
> >
> >
> > Thanks,
> >
> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >
> >
> > On Wed, Feb 19, 2014 at 3:54 PM, Alexander Duyck <
> > alexander.h.du...@intel.com> wrote:
> >
> >> Scott,
> >>
> >> The queue 0 issue and the status of the DCA registers points to the use
> >> of an older driver. The issue was we were initializing the cpu to 0 for
> >> the q_vectors when we were allocating them. As a result the queues that
> >> ended up on CPU zero were not setting the tag correctly. That is why we
> >> moved the intial value to -1 for the q_vector->cpu value in commit
> >> 245f292d71d3fdd7536c2e4986769d5b9b48fb7f. Based on the behavior it
> >> sounds like the driver you have is probably something from before 2013,
> >> if you have a driver with a version number 3.11.X or greater you
> >> shouldn't have the issue.
> >>
> >> Thanks,
> >>
> >> Alex
> >>
> >>
> >> On 02/19/2014 12:49 PM, Scott Silverman wrote:
> >>> Alex,
> >>>
> >>> Thanks for the tip about using ethregs to inspect the DCA registers.
> >>> However, this has led to another question. Specifically, it seems that
> >>> while DCA is enabled for most of the queues, those that land on core 0
> >>> seem to stay disabled for some reason.
> >>>
> >>> Here's a snip from one of my machines with 24 cores (and 24 queues).
> >>> 001-023 all show enabled, but 000 does not:
> >>> [root@sec54 ethregs-1.16.0]# ./ethregs -d 8086:10fb | grep DCA | more
> >>> DCA_RXCTRL[000] 00001200
> >>> DCA_RXCTRL[001] 1a0002a0
> >>> DCA_RXCTRL[002] 190002a0
> >>> DCA_RXCTRL[003] 170002a0
> >>> DCA_RXCTRL[004] 160002a0
> >>> DCA_RXCTRL[005] 150002a0
> >>> DCA_RXCTRL[006] 1b0002a0
> >>> DCA_RXCTRL[007] 1a0002a0
> >>> DCA_RXCTRL[008] 190002a0
> >>> DCA_RXCTRL[009] 170002a0
> >>> DCA_RXCTRL[010] 160002a0
> >>> DCA_RXCTRL[011] 150002a0
> >>> DCA_RXCTRL[012] 1b0002a0
> >>> DCA_RXCTRL[013] 1a0002a0
> >>> DCA_RXCTRL[014] 190002a0
> >>> DCA_RXCTRL[015] 170002a0
> >>> DCA_RXCTRL[016] 160002a0
> >>> DCA_RXCTRL[017] 150002a0
> >>> DCA_RXCTRL[018] 1b0002a0
> >>> DCA_RXCTRL[019] 1a0002a0
> >>> DCA_RXCTRL[020] 190002a0
> >>> DCA_RXCTRL[021] 170002a0
> >>> DCA_RXCTRL[022] 160002a0
> >>> DCA_RXCTRL[023] 150002a0
> >>> DCA_RXCTRL[024] 00001200
> >>> DCA_RXCTRL[025] 00001200
> >>>
> >>> The TXCTRL shows the same pattern.
> >>>
> >>> I'm not sure what to make of this, and I'm also not sure how to get
> >>> DCA running on queue 0.
> >>>
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Scott Silverman | IT | Simplex Investments | 312-360-2444
> >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>>
> >>>
> >>> On Thu, Feb 13, 2014 at 4:18 PM, Alexander Duyck
> >>> <alexander.h.du...@intel.com <mailto:alexander.h.du...@intel.com>>
> >> wrote:
> >>> Odds are the problem is the order the modules are loading in. In
> >>> order
> >>> to print the message ioatdma and dca needs to be loaded before
> >>> ixgbe, if
> >>> the ioatdma driver is loaded after then you won't see the message
> >>> as it
> >>> is only printed at probe time and not when a DCA provider is
> >>> registered.
> >>>
> >>> If you are loading dca, ixgbe, and then ioatdma in that order the
> >>> ixgbe
> >>> won't show as having DCA enabled because the ixgbe_notify_dca call
> >>> doesn't appear to print any message. As a result you would have
> DCA
> >>> enabled, but it won't have displayed any message stating as such.
> >>>
> >>> The tell-tale sign that you have DCA enabled is to use the ethregs
> >>> tool
> >>> to dump the registers for the device. The DCA_RXCTRL and
> DCA_TXCTRL
> >>> registers will have a APIC tag ID in the upper 8 bits. Usually the
> >>> value is something like 1f or 1e. If the value is 00 then it is
> >>> disabled. Below is a snippet of a register dump on a system I
> >>> have here
> >>> that supports DCA.
> >>>
> >>> DCA Enabled:
> >>> DCA_TXCTRL[0] 1f002220
> >>> DCA_TXCTRL[1] 1f002220
> >>>
> >>> DCA Disabled:
> >>> DCA_TXCTRL[0] 00002220
> >>> DCA_TXCTRL[1] 00002220
> >>>
> >>> Thanks,
> >>>
> >>> Alex
> >>>
> >>> On 02/13/2014 01:38 PM, Scott Silverman wrote:
> >>> > It seems I was making a mistake, just not the one I thought I
> was.
> >>> > When I looked at 3.18.7, it was after a system boot. When I
> >>> looked at
> >>> > 3.19.1 it was only after removing and reloading the module.
> >>> >
> >>> > I've attached dmesg output from a system boot just now, showing
> >>> dca /
> >>> > igb / ixgbe / ioatdma modules all loading. It seems that for
> >>> whatever
> >>> > reason, during the system boot DCA is not enabled for ixgbe (but
> >>> it is
> >>> > for igb). If I remove and reload ixgbe, it then enables DCA.
> >>> (seen at
> >>> > the end of the attached dmesg).
> >>> >
> >>> > Is this the expected behavior? Am I doing something wrong? Do I
> >> need
> >>> > to ensure that ioatdma loads before ixgbe? (if so, why doesn't
> igb
> >>> > seem to care?)
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>> >
> >>> >
> >>> > On Thu, Feb 13, 2014 at 3:18 PM, Scott Silverman
> >>> > <ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>
> >>> > <mailto:ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>>> wrote:
> >>> >
> >>> > Alex/John,
> >>> >
> >>> > Thanks for the clarification with regards to DDIO/DCA.
> >>> >
> >>> > As far as my results with the 3.18.7 driver, I can't
> duplicate
> >>> > them now, so I'll chalk it up to a mistake on my side. Sorry
> >> for
> >>> > the trouble.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> >>> <tel:312-360-2444>
> >>> > <tel:312-360-2444 <tel:312-360-2444>>
> >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>> >
> >>> >
> >>> > On Thu, Feb 13, 2014 at 3:03 PM, Duyck, Alexander H
> >>> > <alexander.h.du...@intel.com
> >>> <mailto:alexander.h.du...@intel.com>
> >>> <mailto:alexander.h.du...@intel.com
> >>> <mailto:alexander.h.du...@intel.com>>>
> >>> > wrote:
> >>> >
> >>> > Scott,
> >>> >
> >>> >
> >>> >
> >>> > DDIO does not completely replace DCA. DCA provides
> >>> > functionality for remote socket, while DDIO only
> functions
> >>> > with the local socket for the device.
> >>> >
> >>> >
> >>> >
> >>> > The ixgbe driver prints that DCA message if DCA was
> >> detected
> >>> > when the driver was loaded. Did you try running lsmod
> >>> to see
> >>> > if ioatdma and dca modules were loaded when you loaded
> the
> >>> > 3.18.7 driver?
> >>> >
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> >
> >>> >
> >>> > Alex
> >>> >
> >>> >
> >>> >
> >>> > *From:*Scott Silverman
> >>> > [mailto:ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>
> >>> > <mailto:ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>>]
> >>> > *Sent:* Thursday, February 13, 2014 12:36 PM
> >>> > *To:* Duyck, Alexander H
> >>> > *Cc:* Ronciak, John; e1000-devel@lists.sourceforge.net
> >>> <mailto:e1000-devel@lists.sourceforge.net>
> >>> > <mailto:e1000-devel@lists.sourceforge.net
> >>> <mailto:e1000-devel@lists.sourceforge.net>>
> >>> > *Subject:* Re: [E1000-devel] DCA on Sandy Bridge?
> >>> >
> >>> >
> >>> >
> >>> > John,
> >>> >
> >>> >
> >>> >
> >>> > I think you misunderstood. Sandy Bridge ioatdma/dca
> >>> should be
> >>> > supported in kernel 3.4.41. However, I only see that
> "DCA"
> >>> > flag in ixgbe/dmesg when I am using ixgbe 3.19.1, not
> when
> >> I
> >>> > am using 3.18.7. I am asking why that might be. Nothing
> >> else
> >>> > changed, aside from rmmod ixgbe (3.18.7) and modprobe
> ixgbe
> >>> > (3.19.1).
> >>> >
> >>> >
> >>> >
> >>> > Alex,
> >>> >
> >>> >
> >>> >
> >>> > There is nothing in dmesg to explain it. In fact, the igb
> >>> > driver always uses DCA (prints "DCA enabled"), so I know
> >>> that
> >>> > from a platform perspective it is working.
> >>> >
> >>> >
> >>> >
> >>> > To put it simply, I have two questions:
> >>> >
> >>> >
> >>> >
> >>> > 1. On Sandy bridge (and Ivy Bridge, etc), does DDIO
> replace
> >>> > DCA? If so, what implication does the presence of "DCA"
> in
> >>> > enabled features on a platform that is meant to have DDIO
> >>> > instead of DCA?
> >>> >
> >>> > 2. What change from 3.18.7 to 3.19.1 would cause the
> >> feature
> >>> > to become Enabled on a platform that otherwise supports
> >> DCA?
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> >
> >>> >
> >>> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> >>> > <tel:312-360-2444 <tel:312-360-2444>>
> >>> >
> >>> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Feb 13, 2014 at 1:59 PM, Alexander Duyck
> >>> > <alexander.h.du...@intel.com
> >>> <mailto:alexander.h.du...@intel.com>
> >>> > <mailto:alexander.h.du...@intel.com
> >>> <mailto:alexander.h.du...@intel.com>>> wrote:
> >>> >
> >>> > Scott,
> >>> >
> >>> > You might try checking the dmesg log on the systems that
> >>> come
> >>> > up as not
> >>> > supporting DCA. There was a patch submitted a year or
> >>> so ago
> >>> > that would
> >>> > disable DCA on platforms that had a misconfigured APIC
> >>> ID tag map.
> >>> >
> >>> > It is possible that the platform might have had DCA
> >> disabled
> >>> > due to this
> >>> > in the case of the 3.18.7. You might want to go back and
> >>> > recheck the
> >>> > dmesg log to verify if DCA was disabled due to a
> >>> configuration
> >>> > error in
> >>> > the BIOS.
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Alex
> >>> >
> >>> >
> >>> > On 02/13/2014 08:27 AM, Scott Silverman wrote:
> >>> > > John,
> >>> > >
> >>> > > Thanks for the prompt response, but I still have some
> >>> > questions/concerns.
> >>> > >
> >>> > > -My motherboard (SuperMicro X9DRW) specifically
> provides
> >> a
> >>> > firmware option
> >>> > > to enable "DCA" but you state that the chipset doesn't
> >>> > include it at all?
> >>> > >
> >>> > > -Why is it reported only sometimes?
> >>> > >
> >>> > > Proc, Kern, ixgbe - DCA?
> >>> > > Sandy, 3.4, 3.18.7 - No DCA
> >>> > > Sandy, 3.4, 3.19.1 - DCA
> >>> > > Sandy, 3.12, 3.19.1 - DCA
> >>> > > Ivy, 3.12, 3.18.7 - DCA
> >>> > > Ivy, 3.12, 3.19.1 - DCA
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > Thanks,
> >>> > >
> >>> > > Scott Silverman | IT | Simplex Investments |
> 312-360-2444
> >>> > <tel:312-360-2444 <tel:312-360-2444>>
> >>> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>> > >
> >>> > >
> >>> > > On Mon, Feb 10, 2014 at 4:26 PM, Ronciak, John
> >>> > <john.ronc...@intel.com <mailto:john.ronc...@intel.com>
> >>> <mailto:john.ronc...@intel.com <mailto:john.ronc...@intel.com
> >>>>> wrote:
> >>> > >
> >>> > >> The driver is just telling you what's possible not
> >>> that DCA
> >>> > is enabled.
> >>> > >> The newer chipsets do not included it as it's all
> DDIO.
> >>> > So even though
> >>> > >> the NIC's is capable of support DCA the chipset does
> not
> >>> > have it so it
> >>> > >> won't be used. That's all this is saying. It's not
> an
> >>> > issue at all.
> >>> > >>
> >>> > >> Cheers,
> >>> > >> John
> >>> > >>
> >>> > >>
> >>> > >>> -----Original Message-----
> >>> > >>> From: Scott Silverman
> >>> > [mailto:ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>
> >>> > <mailto:ssilver...@simplexinvestments.com
> >>> <mailto:ssilver...@simplexinvestments.com>>]
> >>> > >>> Sent: Monday, February 10, 2014 11:59 AM
> >>> > >>> To: e1000-devel@lists.sourceforge.net
> >>> <mailto:e1000-devel@lists.sourceforge.net>
> >>> > <mailto:e1000-devel@lists.sourceforge.net
> >>> <mailto:e1000-devel@lists.sourceforge.net>>
> >>> > >>> Subject: [E1000-devel] DCA on Sandy Bridge?
> >>> > >>>
> >>> > >>> My understanding is the DDIO completely replaces DCA
> on
> >>> > sandy bridge
> >>> > >>> and newer hardware (E5+ xeons). I expected this is
> why
> >> I
> >>> > don't see
> >>> > >>> "DCA" listed when I load ixgbe, as I would on my
> >> nehalem
> >>> > systems.
> >>> > >>>
> >>> > >>> I see the following when loading ixgbe 3.18.7 on an
> >>> > E5-2670 system
> >>> > >>> (kernel
> >>> > >>> 3.4.41):
> >>> > >>>
> >>> > >>> ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8)
> >>> > 00:1b:21:5c:66:0d
> >>> > >>> ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4,
> >>> PBA No:
> >>> > E68793-002
> >>> > >>> ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 32
> >> TxQ:
> >>> > 32 FdirHash
> >>> > >>> RSC ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit
> >>> Network
> >>> > Connection
> >>> > >>>
> >>> > >>> If I take the same system, unload ixgbe, and load
> ixgbe
> >>> > 3.19.1, I see
> >>> > >>> this (emphasis mine):
> >>> > >>>
> >>> > >>> ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s
> >>> > available ixgbe
> >>> > >>> 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding
> >>> > Loss:20%) ixgbe
> >>> > >>> 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No:
> >>> > E68793-002 ixgbe
> >>> > >>> 0000:04:00.1: 00:1b:21:5c:66:0d ixgbe 0000:04:00.1:
> >>> eth3:
> >>> > Enabled
> >>> > >>> Features: RxQ: 32 TxQ: 32 FdirHash *DCA*RSC ixgbe
> >>> > 0000:04:00.1: eth3:
> >>> > >>> Intel(R) 10 Gigabit Network Connection
> >>> > >>>
> >>> > >>> Can anyone explain why I see "DCA" with 3.19.1 and
> not
> >>> > with 3.18.7.
> >>> > >>> Also, does it matter?
> >>> > >>>
> >>> > >>> Thanks,
> >>> > >>>
> >>> > >>> Scott Silverman | IT | Simplex Investments |
> >>> 312-360-2444
> >>> > <tel:312-360-2444 <tel:312-360-2444>>
> >>> > >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >>> > >>
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>
>
>
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
