Hi Donald,
Thank you for your reply.
>
> I have a couple additional questions
> - What were you running to do this test? (i.e. netperf, pktgen?)
netperf-2.4.2-1.el5.rf
> - What throughput were you seeing?
On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s
> - Are both ports plugged into x8 PCIe slots
Yes, both servers are Dell R710 with the same hardware configuration
> - Did you look at the dmesg for anything relative?
I see no error or warning messages on both side.
> - Do you see this issue only with UDP?
If I send a TCP stream I see on both side a throughput of about 5.54
Gb/s and less rx_missed_errors on rx side:
NIC statistics:
rx_packets: 1768101
tx_packets: 235384
rx_bytes: 15937257320
tx_bytes: 15704254
lsc_int: 2
tx_busy: 0
non_eop_descs: 7072222
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 1
broadcast: 0
rx_no_buffer_count: 0
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 1539
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_tcp4_seg_ctxt: 0
tx_tcp6_seg_ctxt: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_good: 1768098
rx_csum_offload_errors: 0
tx_csum_offload_ctxt: 11
low_latency_interrupt: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
lro_aggregated: 1760417
lro_flushed: 375277
lro_recycled: 1012791
rx_no_dma_resources: 0
hw_rsc_count: 0
rx_flm: 0
fdir_match: 0
fdir_miss: 0
tx_queue_0_packets: 235384
tx_queue_0_bytes: 15704254
rx_queue_0_packets: 2
rx_queue_0_bytes: 120
rx_queue_1_packets: 8
rx_queue_1_bytes: 786
rx_queue_2_packets: 301752
rx_queue_2_bytes: 2719959436
rx_queue_3_packets: 562919
rx_queue_3_bytes: 5074118826
rx_queue_4_packets: 7
rx_queue_4_bytes: 726
rx_queue_5_packets: 15
rx_queue_5_bytes: 1526
rx_queue_6_packets: 313901
rx_queue_6_bytes: 2829476778
rx_queue_7_packets: 589497
rx_queue_7_bytes: 5313699122
I guess it's because of TCP flow control.
I remember that the same behavior was triggered by enabling flow control
on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to
wait till Monday to perform another run to test that configuration.
Thanks,
Mirko
>
> What I'm wondering is if this NIC was receiving faster than it could process
> them.
>
> Thanks,
> -Don Skidmore <[email protected]>
>
>
>> -----Original Message-----
>> From: Mirko Corosu [mailto:[email protected]]
>> Sent: Friday, February 19, 2010 5:30 AM
>> To: [email protected]
>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet
>>
>> Dear all,
>>
>> I'm new to this list, if this isn't the right place to post this issue,
>> please let me know.
>>
>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on two
>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The two
>> NICs are directly connected each other (no switch in between). I am
>> experiencing a problem on the receive side of the connection: I'm sending
>> an UDP stream after having disabled flow control on each card, and about
>> the 60% of the transmitted packets are dropped with a large number of
>> rx_missed_errors. This is what ifconfig and ethtool -S show:
>>
>> [r...@grids2]# ethtool -S eth0
>> NIC statistics:
>> rx_packets: 1761794
>> tx_packets: 32
>> rx_bytes: 14748667944
>> tx_bytes: 5746
>> lsc_int: 2
>> tx_busy: 0
>> non_eop_descs: 6510828
>> rx_errors: 0
>> tx_errors: 0
>> rx_dropped: 0
>> tx_dropped: 0
>> multicast: 2
>> broadcast: 2
>> rx_no_buffer_count: 0
>> collisions: 0
>> rx_over_errors: 0
>> rx_crc_errors: 0
>> rx_frame_errors: 0
>> rx_fifo_errors: 0
>> rx_missed_errors: 1148382
>> tx_aborted_errors: 0
>> tx_carrier_errors: 0
>> tx_fifo_errors: 0
>> tx_heartbeat_errors: 0
>> tx_timeout_count: 0
>> tx_restart_queue: 0
>> rx_long_length_errors: 0
>> rx_short_length_errors: 0
>> tx_tcp4_seg_ctxt: 0
>> tx_tcp6_seg_ctxt: 0
>> tx_flow_control_xon: 0
>> rx_flow_control_xon: 0
>> tx_flow_control_xoff: 0
>> rx_flow_control_xoff: 0
>> rx_csum_offload_good: 28
>> rx_csum_offload_errors: 0
>> tx_csum_offload_ctxt: 11
>> low_latency_interrupt: 0
>> alloc_rx_page_failed: 0
>> alloc_rx_buff_failed: 0
>> lro_aggregated: 8
>> lro_flushed: 8
>> lro_recycled: 0
>> rx_no_dma_resources: 0
>> hw_rsc_count: 0
>> rx_flm: 0
>> fdir_match: 0
>> fdir_miss: 0
>> tx_queue_0_packets: 32
>> tx_queue_0_bytes: 5746
>> rx_queue_0_packets: 2
>> rx_queue_0_bytes: 120
>> rx_queue_1_packets: 2
>> rx_queue_1_bytes: 120
>> rx_queue_2_packets: 1761762
>> rx_queue_2_bytes: 14748664800
>> rx_queue_3_packets: 0
>> rx_queue_3_bytes: 0
>> rx_queue_4_packets: 0
>> rx_queue_4_bytes: 0
>> rx_queue_5_packets: 14
>> rx_queue_5_bytes: 1452
>> rx_queue_6_packets: 7
>> rx_queue_6_bytes: 726
>> rx_queue_7_packets: 7
>> rx_queue_7_bytes: 726
>>
>> [r...@grids2]# ifconfig eth0
>>
>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF
>> inet addr:10.100.100.2 Bcast:10.100.100.255 Mask:255.255.255.0
>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 frame:0
>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB)
>>
>> Each interface has the MTU set to 9000, the driver is the one provided
>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled and
>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too).
>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no success.
>> Single CPU load is no more than 10% with 8000 interrupt/s:
>>
>> [r...@grids2 ~]# mpstat -P ALL 2
>>
>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft %steal
>> %idle intr/s
>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 0.00
>> 98.69 9018.00
>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1002.50
>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1.50
>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 0.00
>> 89.55 8007.00
>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1.00
>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1.00
>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 3.00
>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1.00
>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 100.00 1.00
>>
>> I tried to configure smp affinity to assign the IRQ of each
>> rx-queue to a different CPU, with no effect on the amount of dropped
>> packets. I also verified that if I connect the NIC on a 10GE switch and
>> send more than one stream from different servers, the card uses
>> different rx-queues but the problem still exists.
>>
>> I modified the /etc/sysctl.conf file parameters adding:
>>
>> net.core.wmem_max = 16777216
>> net.core.wmem_default = 8388608
>> net.core.rmem_max = 16777216
>> net.core.rmem_default = 8388608
>> net.ipv4.tcp_rmem = 4096 262144 16777216
>> net.ipv4.tcp_wmem = 4096 262144 16777216
>> net.ipv4.tcp_mem = 4096 8388608 16777216
>> net.core.optmem_max = 524288
>> net.core.netdev_max_backlog = 200000
>>
>>
>>
>>
>> Some other details:
>>
>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading disabled)
>> - RAM: 24 GB (12 X 2GB)
>> - Slot PCI: PCIe gen.2 x8
>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5
>>
>> Any suggestions?
>> Thank you very much in advance.
>>
>> Mirko Corosu
>>
>> --
>>
>> --------------------------------------------------------------------------
>> Mirko Corosu
>> Network and system administrator
>> Computing Center
>> Istituto Nazionale Fisica Nucleare
>> Via Dodecaneso 33
>> 16146 Genova, Italy
>> http://www.ge.infn.it
>> --------------------------------------------------------------------------
>>
>>
>>
>> ---------------------------------------------------------------------------
>> ---
>> Download Intel® Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> E1000-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/e1000-devel
>> To learn more about Intel® Ethernet, visit
>> http://communities.intel.com/community/wired
--
--------------------------------------------------------------------------
Mirko Corosu
Istituto Nazionale Fisica Nucleare
Via Dodecaneso 33
16146 Genova
www.ge.infn.it
Phone +39 010 3536361
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata.
--------------------------------------------------------------------------
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
