On 08/12/2013 12:09 PM, Alexey Stoyanov wrote: > Hello > I got one issue, and seems i need help from driver developers. > > I have a some servers located in a different datacenters around > Russia, we used mostly 82575/827576 intel nic managed by e1000e and > igb drivers. When i testing speed with iperf from one 82576 card to > another - all working good, i can get 400-500-800mbit/s with one tcp > stream. > > In last 5-6 months we bought some 10G intel 82599 cards and i see some > really strange issue, speed of tcp stream via 10G port is much slower > (from 3 to 10 times) than the 1G 82575/82576 link. > > I have same server, same OS, same kernel - just difference is eth1 a > 82575 (e1000e), and eth2 a 82599. > > ethtool -i eth1 > driver: ixgbe > version: 3.17.3 > firmware-version: 0x61c10001 > bus-info: 0000:03:00.0 > > ethtool -i eth2 > driver: e1000e > version: 2.1.4-k > firmware-version: 1.8-0 > bus-info: 0000:07:00.0 > > i bind different ip to each port > eth1 Link encap:Ethernet HWaddr 90:e2:ba:40:89:24 > inet addr:xxx.xxx.185.135 Bcast:xxx.xxx.185.143 > Mask:255.255.255.240 > > eth2 Link encap:Ethernet HWaddr 00:25:90:09:68:1b > inet addr:xxx.xxx.185.133 Bcast:xxx.xxx.185.143 > Mask:255.255.255.240 > > and tried with iperf > > this is 82575 fiber > ------------------------------------------------------------ > Client connecting to yyy.yyy.74.11, TCP port 5001 > Binding to local address xxx.xxx.185.133 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local xxx.xxx.185.133 port 5001 connected with yyy.yyy.74.11 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.0 sec 1.03 GBytes 443 Mbits/sec > > > and this is 82599 optic > ------------------------------------------------------------ > Client connecting to yyy.yyy.74.11, TCP port 5001 > Binding to local address xxx.xxx.185.135 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local xxx.xxx.185.135 port 5001 connected with yyy.yyy.74.11 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.1 sec 363 MBytes 151 Mbits/sec > > The difference is impressive, cooper 82575 with e1000e done 3 times > faster than optical 82599. > Looks like ixgbe driver work in different way with tcp_window or maybe > tcp_congestion, because in realtime monitoring on 82575 i can see how > speed raise every second from 10MB/s to 90-100MB/s with 5-8MB/s step. > But on 82599 it just flowing around 10MB/s and won't go up. > > i used this tuning to reach up to 900mbit > > net.ipv4.tcp_rmem = 4096 87530 33554432 > net.ipv4.tcp_wmem = 4096 65535 33554432 > net.core.netdev_max_backlog = 3000 > net.ipv4.tcp_congestion_control = htcp > net.ipv4.tcp_timestamps = 1 > net.ipv4.tcp_reordering = 100 > net.ipv4.tcp_dsack = 0 > net.ipv4.tcp_sack = 1 > net.ipv4.tcp_fack = 0 > > I use > Linux 3.8.0-26-generic #38-Ubuntu SMP Mon Jun 17 21:43:33 UTC 2013 > x86_64 x86_64 x86_64 GNU/Linux > Ubuntu 13.04 > > I ready to give any additional information and do any test. > I tried last driver from SF, turn off and on all offload, even tried > turn off interrupt moderation and set ring buffer to 64 to serve > traffic faster. > > Both cards attached to same switch with 10G uplink. > >
Could you include a copy of the dmesg log, the lspci -vvv, and ethtool -S output for the 82599 and the 82575 parts? Thanks, Alex ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
