Hey Mirko, Thanks for your reply. I have some additional clarifications on some of your answers.
>> I have a couple additional questions >> - What were you running to do this test? (i.e. netperf, pktgen?) > >netperf-2.4.2-1.el5.rf How were you using netperf in your tests? What tests (I assume [UDP|TCP]_STREAM) and how many instances of netperf were running each side? >> - Are both ports plugged into x8 PCIe slots > >Yes, both servers are Dell R710 with the same hardware configuration >From the spec I got off the web on the R710 I can see they have either (2 PCIe >x8 and 2 PCIe x4) or (1 PCIe x16 and 2 PCIe x4 ). I was just concerned you >might be plugged into one of the x4 slots. >> - Did you look at the dmesg for anything relative? > >I see no error or warning messages on both side. It's not just error or warning messages I was interested in, although they would be interesting too. :) We also printk a message that displays the type of PCIe link we have. This would answer the question I posed above about what slot the cards were in. All that said we would expect to see rx_missed_errors with flow control disabled. Since nothing would be stopping packets from overrunning the buffer. Your test seems to be agreeing with this as the number of rx_missed_errors goes down with TCP (like you said) most likely due to TCP sliding window. You don't see this issue with flow control enabled, right? Thanks, -Don >-----Original Message----- >From: Mirko Corosu [mailto:[email protected]] >Sent: Saturday, February 20, 2010 10:50 AM >To: Skidmore, Donald C >Cc: [email protected] >Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet > >Hi Donald, > >Thank you for your reply. > >> >> I have a couple additional questions >> - What were you running to do this test? (i.e. netperf, pktgen?) > >netperf-2.4.2-1.el5.rf > >> - What throughput were you seeing? > >On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s > >> - Are both ports plugged into x8 PCIe slots > >Yes, both servers are Dell R710 with the same hardware configuration > >> - Did you look at the dmesg for anything relative? > >I see no error or warning messages on both side. > >> - Do you see this issue only with UDP? > >If I send a TCP stream I see on both side a throughput of about 5.54 >Gb/s and less rx_missed_errors on rx side: > >NIC statistics: > rx_packets: 1768101 > tx_packets: 235384 > rx_bytes: 15937257320 > tx_bytes: 15704254 > lsc_int: 2 > tx_busy: 0 > non_eop_descs: 7072222 > rx_errors: 0 > tx_errors: 0 > rx_dropped: 0 > tx_dropped: 0 > multicast: 1 > broadcast: 0 > rx_no_buffer_count: 0 > collisions: 0 > rx_over_errors: 0 > rx_crc_errors: 0 > rx_frame_errors: 0 > rx_fifo_errors: 0 > rx_missed_errors: 1539 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > tx_timeout_count: 0 > tx_restart_queue: 0 > rx_long_length_errors: 0 > rx_short_length_errors: 0 > tx_tcp4_seg_ctxt: 0 > tx_tcp6_seg_ctxt: 0 > tx_flow_control_xon: 0 > rx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > rx_flow_control_xoff: 0 > rx_csum_offload_good: 1768098 > rx_csum_offload_errors: 0 > tx_csum_offload_ctxt: 11 > low_latency_interrupt: 0 > alloc_rx_page_failed: 0 > alloc_rx_buff_failed: 0 > lro_aggregated: 1760417 > lro_flushed: 375277 > lro_recycled: 1012791 > rx_no_dma_resources: 0 > hw_rsc_count: 0 > rx_flm: 0 > fdir_match: 0 > fdir_miss: 0 > tx_queue_0_packets: 235384 > tx_queue_0_bytes: 15704254 > rx_queue_0_packets: 2 > rx_queue_0_bytes: 120 > rx_queue_1_packets: 8 > rx_queue_1_bytes: 786 > rx_queue_2_packets: 301752 > rx_queue_2_bytes: 2719959436 > rx_queue_3_packets: 562919 > rx_queue_3_bytes: 5074118826 > rx_queue_4_packets: 7 > rx_queue_4_bytes: 726 > rx_queue_5_packets: 15 > rx_queue_5_bytes: 1526 > rx_queue_6_packets: 313901 > rx_queue_6_bytes: 2829476778 > rx_queue_7_packets: 589497 > rx_queue_7_bytes: 5313699122 > >I guess it's because of TCP flow control. > >I remember that the same behavior was triggered by enabling flow control >on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to >wait till Monday to perform another run to test that configuration. > >Thanks, > >Mirko > >> >> What I'm wondering is if this NIC was receiving faster than it could >process them. >> >> Thanks, >> -Don Skidmore <[email protected]> >> >> >>> -----Original Message----- >>> From: Mirko Corosu [mailto:[email protected]] >>> Sent: Friday, February 19, 2010 5:30 AM >>> To: [email protected] >>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>> >>> Dear all, >>> >>> I'm new to this list, if this isn't the right place to post this issue, >>> please let me know. >>> >>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on two >>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The two >>> NICs are directly connected each other (no switch in between). I am >>> experiencing a problem on the receive side of the connection: I'm >sending >>> an UDP stream after having disabled flow control on each card, and about >>> the 60% of the transmitted packets are dropped with a large number of >>> rx_missed_errors. This is what ifconfig and ethtool -S show: >>> >>> [r...@grids2]# ethtool -S eth0 >>> NIC statistics: >>> rx_packets: 1761794 >>> tx_packets: 32 >>> rx_bytes: 14748667944 >>> tx_bytes: 5746 >>> lsc_int: 2 >>> tx_busy: 0 >>> non_eop_descs: 6510828 >>> rx_errors: 0 >>> tx_errors: 0 >>> rx_dropped: 0 >>> tx_dropped: 0 >>> multicast: 2 >>> broadcast: 2 >>> rx_no_buffer_count: 0 >>> collisions: 0 >>> rx_over_errors: 0 >>> rx_crc_errors: 0 >>> rx_frame_errors: 0 >>> rx_fifo_errors: 0 >>> rx_missed_errors: 1148382 >>> tx_aborted_errors: 0 >>> tx_carrier_errors: 0 >>> tx_fifo_errors: 0 >>> tx_heartbeat_errors: 0 >>> tx_timeout_count: 0 >>> tx_restart_queue: 0 >>> rx_long_length_errors: 0 >>> rx_short_length_errors: 0 >>> tx_tcp4_seg_ctxt: 0 >>> tx_tcp6_seg_ctxt: 0 >>> tx_flow_control_xon: 0 >>> rx_flow_control_xon: 0 >>> tx_flow_control_xoff: 0 >>> rx_flow_control_xoff: 0 >>> rx_csum_offload_good: 28 >>> rx_csum_offload_errors: 0 >>> tx_csum_offload_ctxt: 11 >>> low_latency_interrupt: 0 >>> alloc_rx_page_failed: 0 >>> alloc_rx_buff_failed: 0 >>> lro_aggregated: 8 >>> lro_flushed: 8 >>> lro_recycled: 0 >>> rx_no_dma_resources: 0 >>> hw_rsc_count: 0 >>> rx_flm: 0 >>> fdir_match: 0 >>> fdir_miss: 0 >>> tx_queue_0_packets: 32 >>> tx_queue_0_bytes: 5746 >>> rx_queue_0_packets: 2 >>> rx_queue_0_bytes: 120 >>> rx_queue_1_packets: 2 >>> rx_queue_1_bytes: 120 >>> rx_queue_2_packets: 1761762 >>> rx_queue_2_bytes: 14748664800 >>> rx_queue_3_packets: 0 >>> rx_queue_3_bytes: 0 >>> rx_queue_4_packets: 0 >>> rx_queue_4_bytes: 0 >>> rx_queue_5_packets: 14 >>> rx_queue_5_bytes: 1452 >>> rx_queue_6_packets: 7 >>> rx_queue_6_bytes: 726 >>> rx_queue_7_packets: 7 >>> rx_queue_7_bytes: 726 >>> >>> [r...@grids2]# ifconfig eth0 >>> >>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF >>> inet addr:10.100.100.2 Bcast:10.100.100.255 >Mask:255.255.255.0 >>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 >>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 frame:0 >>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:1000 >>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB) >>> >>> Each interface has the MTU set to 9000, the driver is the one provided >>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled and >>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too). >>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no success. >>> Single CPU load is no more than 10% with 8000 interrupt/s: >>> >>> [r...@grids2 ~]# mpstat -P ALL 2 >>> >>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft %steal >>> %idle intr/s >>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 0.00 >>> 98.69 9018.00 >>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1002.50 >>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1.50 >>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 0.00 >>> 89.55 8007.00 >>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1.00 >>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1.00 >>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 3.00 >>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1.00 >>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 100.00 1.00 >>> >>> I tried to configure smp affinity to assign the IRQ of each >>> rx-queue to a different CPU, with no effect on the amount of dropped >>> packets. I also verified that if I connect the NIC on a 10GE switch and >>> send more than one stream from different servers, the card uses >>> different rx-queues but the problem still exists. >>> >>> I modified the /etc/sysctl.conf file parameters adding: >>> >>> net.core.wmem_max = 16777216 >>> net.core.wmem_default = 8388608 >>> net.core.rmem_max = 16777216 >>> net.core.rmem_default = 8388608 >>> net.ipv4.tcp_rmem = 4096 262144 16777216 >>> net.ipv4.tcp_wmem = 4096 262144 16777216 >>> net.ipv4.tcp_mem = 4096 8388608 16777216 >>> net.core.optmem_max = 524288 >>> net.core.netdev_max_backlog = 200000 >>> >>> >>> >>> >>> Some other details: >>> >>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading disabled) >>> - RAM: 24 GB (12 X 2GB) >>> - Slot PCI: PCIe gen.2 x8 >>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5 >>> >>> Any suggestions? >>> Thank you very much in advance. >>> >>> Mirko Corosu >>> >>> -- >>> >>> ------------------------------------------------------------------------ >-- >>> Mirko Corosu >>> Network and system administrator >>> Computing Center >>> Istituto Nazionale Fisica Nucleare >>> Via Dodecaneso 33 >>> 16146 Genova, Italy >>> http://www.ge.infn.it >>> ------------------------------------------------------------------------ >-- >>> >>> >>> >>> ------------------------------------------------------------------------ >--- >>> --- >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> E1000-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>> To learn more about Intel® Ethernet, visit >>> http://communities.intel.com/community/wired > > >-- > >-------------------------------------------------------------------------- >Mirko Corosu >Istituto Nazionale Fisica Nucleare >Via Dodecaneso 33 >16146 Genova >www.ge.infn.it >Phone +39 010 3536361 >-------------------------------------------------------------------------- > >-------------------------------------------------------------------------- >Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >-------------------------------------------------------------------------- ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
