On 02/18/2014 07:45 PM, Skidmore, Donald C wrote:
>> -----Original Message-----
>> From: John-Paul Robinson [mailto:j...@uab.edu]
>> Sent: Tuesday, February 18, 2014 2:59 PM
>> To: Brandeburg, Jesse
>> Cc: e1000-devel@lists.sourceforge.net
>> Subject: Re: [E1000-devel] questions on ixgbe and 10G performance
>> expectations
>>
>> On 02/17/2014 08:19 PM, Brandeburg, Jesse wrote:
>>> Forgive my top post.
>>>
>>> With the new kernel you may be running into needing faster cleanup to
>> increase tx speed. try increasing the interrupt rate via ethtool -C ethX rx-
>> usecs 10, yes I said rx because there is only one rate control for the
>> interrupt.
>>>
>>> You can easily do line rate tx with 82599. The biggest limiter in tx only
>>> tests
>> is the amount of data in flight and the time it takes to get acks back.
>>>
>>
>> Thanks for the suggestions.
>>
>> So I tried upping the rx-usecs on the server install instance to 10
>> (originally 1)
>> and saw a clear bump up to what I would consider line rate ~9.36Gbs.
>> Switching to 10 usecs sounds like its a decrease in interrupt rate though.
>>
>> Interestingly I tested my live iso version of ubuntu 12.04.4 desktop again
>> and
>> see an ~9.39Gbs line rate with out any tuning (default ixgbe driver 3.13.10,
>> default rx-usecs@1, same upstream iperf server).
>> Switching to the rx-usecs=10 on this platform degraded the performance, to
>> 8.69Gbs.
>>
>> Ubuntu no longer maintains separate desktop and server kernels, so I'm
>> trusting all the core-kernel operation would be identical. Thus the live iso
>> test is likely as pristine an experience as can be had, wrt stock
>> performance.
>> It'd take it if I could get it. ;)
>>
>> I'd take from this that there is some functionality or setting introduce in
>> an
>> actual system install that's introducing a hit on performance.
>> Any thoughts?
>>
>>> Also please make sure you have run the set_irq_affinity script to bind
>> interrupts to CPUs.
>>>
>>
>> I tried running `set_irg_affinity eth4` but it didn't appear to have any
>> impact
>> on performance. If anything it degraded.
>>
>>> --
>>> Jesse Brandeburg
>>>
>>>
>>>> On Feb 17, 2014, at 5:42 PM, "Ben Greear" <gree...@candelatech.com>
>> wrote:
>>>>
>>>>> On 02/17/2014 02:19 PM, John-Paul Robinson wrote:
>>>>> Hi,
>>>>>
>>>>> I don't know if this topic is appropriate here, please direct me to
>>>>> a better place if not.
>>>>>
>>>>> I've been spending considerable time trying to measure the
>>>>> performance of our 10G fabric that uses Intel X520 cards. The
>>>>> primary test machine has dual Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>> 2.00GHz chip 8-core chips and 96GB RAM.
>>>>>
>>>>> The test machine is now running Ubuntu 12.04.4 server with kernel
>>>>> 3.11.0 with latest ixgbe driver 3.19.1.
>>>>>
>>>>> Using iperf (2.0.5) I see about 9.39Gbs steady inbound transfers
>>>>> (there are a few glitches where I've seen drop to 7Gbs but it
>>>>> recovers). My outbound transfers, however, are about 8.83Gbs steady
>>>>> and tend to be more variable.
>>>>>
>>>>> This is the best performance I can get on the server.
>>>>>
>>>>> Interestingly when I boot the machine off the live CDROM image for
>>>>> Ubuntu 12.04.4 desktop, I see nice steady 9.39Gbs in both directions.
>>>>> This is the best performance i have seen with this card to-date.
>>>>>
>>>>> I've spent a lot of time with these cards and in general they have
>>>>> be very finicky, delivering inconsistent results from test to test,
>>>>> being very sensitive to driver and kernel versions.
>>>>>
>>>>> I've taken them from extremely erratic performance on Ubuntu 12.04.1
>>>>> with the stock ixgbe 3.6.7 driver to much higher, more stable
>>>>> performance simply by updating to ixgbe 3.11.33. It would be nice
>>>>> to see a stable flatline performance at line speeds on kernel 3.11
>>>>> with the
>>>>> 3.19.1 driver.
>>>>>
>>>>> I'm wondering if there is a known configuration profile that allows
>>>>> these cards to perform at line speeds or if there are known issues
>>>>> or hardware incompatibilities.
>>>>>
>>>>> I know there are a lot of subtleties to performance tuning but
>>>>> performance on other cards in our fabric (btw from Brocade) deliver
>>>>> very consistent, stable, high performance line speed results over many
>> tests.
>>>>>
>>>>> I've been scratching my head for a while and am looking for a fresh
>>>>> perspective or deeper understanding.
>>>>
>>>> First, check 'dmesg' and make sure your NICs are using at least
>>>> x8 pci with 5GT/s.
>>>>
>>>> Check BIOS and disable 'VT-d' if it is on...it hurts performance by
>>>> 50% or so.
>>>>
>>>> Try using several (5-10) flows in iperf, maybe just use 5-10
>>>> instances of iperf so you get good usage of your cores.
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>>
>>>> --
>>>> Ben Greear <gree...@candelatech.com>
>>>> Candela Technologies Inc http://www.candelatech.com
>
>
> I would be interested in seeing how your interrupt are being spread out
> during your test. Could you provide the delta from /proc/interrupts before
> and after your test, or just the results after a reboot for the port seeing
> the traffic?
>
> Likewise it would be good to see the delta of the ethtool -S statists from
> before and after your test.
I've attached a tarball with the relevant pre and post test data from
the Ubuntu 12.04.4 live iso "pristine" test. This is the data that
seems to be performing at a pretty solid line-rate without tuning.
Note: this uses the stock ixgbe driver with kernel 3.11.0, v 3.13.10.
>From a look at the interrupts data, at least the second column after the
interrupt number, it looks like the first registered interrupts get the
interrupts. It's not clear to me how to interrupt the subsequent columns.
1c1
< 133: 213905
---
> 133: 214048
34c34
< 166: 65791
---
> 166: 65915
I've also attached two tests from my server based install. One test
with the default rx-usecs of 1 which performs below line-rate on
transmit (8.66Gbs). The second is with the setting of rx-uses to 10
which then performs at close to line-rate (9.30Gbs). Note: this system
has been updated to use the 3.19.1 version of ixgbe. If need be I can
downgrade to the stock ixgbe driver for kernel 3.11.0.
~jpr
>
> Thanks,
> -Don Skidmore <donald.c.skidm...@intel.com>
>
>
>
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
