Hi Donald,
> One thing I did notice was that the Receive side has 8 rx queues while the > Transmit has 16 rx queues. You mentioned these were identical machines could > Hyper-Threading be off on the Receiving machine? > Yes, it was my fault. I forgot to disable hyperthreading on TX server. Now it is disabled, but the problem still remains. > Also I'm still a little confused about how your test is set up. If you have > four netperf's running on, as you called it, your Tx machine and your Rx is > only running netserver then I would only expect heavy traffic from Tx -> Rx. > Or am I missing some thing here? > I try to summarize my tests. For now I am testing only mono-directional TCP and UDP stream (from TX server to RX server directly ). What I cannot understand is the low throughput that I'm getting. The TCP tests shows a throughput of only 5.6 Gb/s. During UDP tests, with flow control disabled, I measured on TX side a throughput of about 9.5 Gb/s (so the card can send at wire speed) but on RX side I see only 5.6 Gb/s. I attribute this problem to the high number of rx_missed_errors but I cannot figure out what is the real cause (I don't know even what rx_missed_errors actually mean....). Thanks Mirko >> -----Original Message----- >> From: Mirko Corosu [mailto:[email protected]] >> Sent: Tuesday, February 23, 2010 3:30 AM >> To: Skidmore, Donald C >> Cc: [email protected] >> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >> >> Hi Donald, >> >> Ok, sorry for the lack of informations: I'm not used to interact with >> developers. :-) >> >> >>> How were you using netperf in your tests? What tests (I assume >> [UDP|TCP]_STREAM) and how many instances of netperf were running each side? >> I'm running four instances of UDP_STREAM or TCP_STREAM tests on the tx >> side, with different packet sizes (from 1000 to 9000 byte) and a single >> netserver on the rx side. >> >>> From the spec I got off the web on the R710 I can see they have either (2 >> PCIe x8 and 2 PCIe x4) or (1 PCIe x16 and 2 PCIe x4 ). I was just >> concerned you might be plugged into one of the x4 slots. >> >> I have 2 PCIe x8 on the so called "Riser 2" and 2 PCIe x4 on the "Riser >> 1". On each server the Intel cards are plugged into the Riser 2 slots. >> >>>>> - Did you look at the dmesg for anything relative? >>>> I see no error or warning messages on both side. >>> It's not just error or warning messages I was interested in, although >> they would be interesting too. :) We also printk a message that displays >> the type of PCIe link we have. This would answer the question I posed >> above about what slot the cards were in. >> On rx side: >> >> [r...@grids2 ~]# dmesg |grep ixgbe >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> None >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> None >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> None >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 8, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> >> >> >> On tx side: >> >> [r...@client20 ~]# dmesg |grep ixgbe >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 16, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >> for DCA functionality. >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> None >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 16, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >> for DCA functionality. >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 16, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >> for DCA functionality. >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> ixgbe: eth0: ixgbe_remove: complete >> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >> 2.0.44.14-NAPI >> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >> Queue count = 16, Tx Queue count = 1 >> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >> for DCA functionality. >> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >> RX/TX >> >> >> >>> All that said we would expect to see rx_missed_errors with flow control >> disabled. Since nothing would be stopping packets from overrunning the >> buffer. Your test seems to be agreeing with this as the number of >> rx_missed_errors goes down with TCP (like you said) most likely due to TCP >> sliding window. >>> You don't see this issue with flow control enabled, right? >> Right, but the throughput remains about 5.6 Gb/s and a can't figure out >> where the bottleneck could be.... >> >> Thanks a lot >> >> Mirko >> >>>> -----Original Message----- >>>> From: Mirko Corosu [mailto:[email protected]] >>>> Sent: Saturday, February 20, 2010 10:50 AM >>>> To: Skidmore, Donald C >>>> Cc: [email protected] >>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>> >>>> Hi Donald, >>>> >>>> Thank you for your reply. >>>> >>>>> I have a couple additional questions >>>>> - What were you running to do this test? (i.e. netperf, pktgen?) >>>> netperf-2.4.2-1.el5.rf >>>> >>>>> - What throughput were you seeing? >>>> On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s >>>> >>>>> - Are both ports plugged into x8 PCIe slots >>>> Yes, both servers are Dell R710 with the same hardware configuration >>>> >>>>> - Did you look at the dmesg for anything relative? >>>> I see no error or warning messages on both side. >>>> >>>>> - Do you see this issue only with UDP? >>>> If I send a TCP stream I see on both side a throughput of about 5.54 >>>> Gb/s and less rx_missed_errors on rx side: >>>> >>>> NIC statistics: >>>> rx_packets: 1768101 >>>> tx_packets: 235384 >>>> rx_bytes: 15937257320 >>>> tx_bytes: 15704254 >>>> lsc_int: 2 >>>> tx_busy: 0 >>>> non_eop_descs: 7072222 >>>> rx_errors: 0 >>>> tx_errors: 0 >>>> rx_dropped: 0 >>>> tx_dropped: 0 >>>> multicast: 1 >>>> broadcast: 0 >>>> rx_no_buffer_count: 0 >>>> collisions: 0 >>>> rx_over_errors: 0 >>>> rx_crc_errors: 0 >>>> rx_frame_errors: 0 >>>> rx_fifo_errors: 0 >>>> rx_missed_errors: 1539 >>>> tx_aborted_errors: 0 >>>> tx_carrier_errors: 0 >>>> tx_fifo_errors: 0 >>>> tx_heartbeat_errors: 0 >>>> tx_timeout_count: 0 >>>> tx_restart_queue: 0 >>>> rx_long_length_errors: 0 >>>> rx_short_length_errors: 0 >>>> tx_tcp4_seg_ctxt: 0 >>>> tx_tcp6_seg_ctxt: 0 >>>> tx_flow_control_xon: 0 >>>> rx_flow_control_xon: 0 >>>> tx_flow_control_xoff: 0 >>>> rx_flow_control_xoff: 0 >>>> rx_csum_offload_good: 1768098 >>>> rx_csum_offload_errors: 0 >>>> tx_csum_offload_ctxt: 11 >>>> low_latency_interrupt: 0 >>>> alloc_rx_page_failed: 0 >>>> alloc_rx_buff_failed: 0 >>>> lro_aggregated: 1760417 >>>> lro_flushed: 375277 >>>> lro_recycled: 1012791 >>>> rx_no_dma_resources: 0 >>>> hw_rsc_count: 0 >>>> rx_flm: 0 >>>> fdir_match: 0 >>>> fdir_miss: 0 >>>> tx_queue_0_packets: 235384 >>>> tx_queue_0_bytes: 15704254 >>>> rx_queue_0_packets: 2 >>>> rx_queue_0_bytes: 120 >>>> rx_queue_1_packets: 8 >>>> rx_queue_1_bytes: 786 >>>> rx_queue_2_packets: 301752 >>>> rx_queue_2_bytes: 2719959436 >>>> rx_queue_3_packets: 562919 >>>> rx_queue_3_bytes: 5074118826 >>>> rx_queue_4_packets: 7 >>>> rx_queue_4_bytes: 726 >>>> rx_queue_5_packets: 15 >>>> rx_queue_5_bytes: 1526 >>>> rx_queue_6_packets: 313901 >>>> rx_queue_6_bytes: 2829476778 >>>> rx_queue_7_packets: 589497 >>>> rx_queue_7_bytes: 5313699122 >>>> >>>> I guess it's because of TCP flow control. >>>> >>>> I remember that the same behavior was triggered by enabling flow control >>>> on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to >>>> wait till Monday to perform another run to test that configuration. >>>> >>>> Thanks, >>>> >>>> Mirko >>>> >>>>> What I'm wondering is if this NIC was receiving faster than it could >>>> process them. >>>>> Thanks, >>>>> -Don Skidmore <[email protected]> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Mirko Corosu [mailto:[email protected]] >>>>>> Sent: Friday, February 19, 2010 5:30 AM >>>>>> To: [email protected] >>>>>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>>> >>>>>> Dear all, >>>>>> >>>>>> I'm new to this list, if this isn't the right place to post this >> issue, >>>>>> please let me know. >>>>>> >>>>>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on >> two >>>>>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The two >>>>>> NICs are directly connected each other (no switch in between). I am >>>>>> experiencing a problem on the receive side of the connection: I'm >>>> sending >>>>>> an UDP stream after having disabled flow control on each card, and >> about >>>>>> the 60% of the transmitted packets are dropped with a large number of >>>>>> rx_missed_errors. This is what ifconfig and ethtool -S show: >>>>>> >>>>>> [r...@grids2]# ethtool -S eth0 >>>>>> NIC statistics: >>>>>> rx_packets: 1761794 >>>>>> tx_packets: 32 >>>>>> rx_bytes: 14748667944 >>>>>> tx_bytes: 5746 >>>>>> lsc_int: 2 >>>>>> tx_busy: 0 >>>>>> non_eop_descs: 6510828 >>>>>> rx_errors: 0 >>>>>> tx_errors: 0 >>>>>> rx_dropped: 0 >>>>>> tx_dropped: 0 >>>>>> multicast: 2 >>>>>> broadcast: 2 >>>>>> rx_no_buffer_count: 0 >>>>>> collisions: 0 >>>>>> rx_over_errors: 0 >>>>>> rx_crc_errors: 0 >>>>>> rx_frame_errors: 0 >>>>>> rx_fifo_errors: 0 >>>>>> rx_missed_errors: 1148382 >>>>>> tx_aborted_errors: 0 >>>>>> tx_carrier_errors: 0 >>>>>> tx_fifo_errors: 0 >>>>>> tx_heartbeat_errors: 0 >>>>>> tx_timeout_count: 0 >>>>>> tx_restart_queue: 0 >>>>>> rx_long_length_errors: 0 >>>>>> rx_short_length_errors: 0 >>>>>> tx_tcp4_seg_ctxt: 0 >>>>>> tx_tcp6_seg_ctxt: 0 >>>>>> tx_flow_control_xon: 0 >>>>>> rx_flow_control_xon: 0 >>>>>> tx_flow_control_xoff: 0 >>>>>> rx_flow_control_xoff: 0 >>>>>> rx_csum_offload_good: 28 >>>>>> rx_csum_offload_errors: 0 >>>>>> tx_csum_offload_ctxt: 11 >>>>>> low_latency_interrupt: 0 >>>>>> alloc_rx_page_failed: 0 >>>>>> alloc_rx_buff_failed: 0 >>>>>> lro_aggregated: 8 >>>>>> lro_flushed: 8 >>>>>> lro_recycled: 0 >>>>>> rx_no_dma_resources: 0 >>>>>> hw_rsc_count: 0 >>>>>> rx_flm: 0 >>>>>> fdir_match: 0 >>>>>> fdir_miss: 0 >>>>>> tx_queue_0_packets: 32 >>>>>> tx_queue_0_bytes: 5746 >>>>>> rx_queue_0_packets: 2 >>>>>> rx_queue_0_bytes: 120 >>>>>> rx_queue_1_packets: 2 >>>>>> rx_queue_1_bytes: 120 >>>>>> rx_queue_2_packets: 1761762 >>>>>> rx_queue_2_bytes: 14748664800 >>>>>> rx_queue_3_packets: 0 >>>>>> rx_queue_3_bytes: 0 >>>>>> rx_queue_4_packets: 0 >>>>>> rx_queue_4_bytes: 0 >>>>>> rx_queue_5_packets: 14 >>>>>> rx_queue_5_bytes: 1452 >>>>>> rx_queue_6_packets: 7 >>>>>> rx_queue_6_bytes: 726 >>>>>> rx_queue_7_packets: 7 >>>>>> rx_queue_7_bytes: 726 >>>>>> >>>>>> [r...@grids2]# ifconfig eth0 >>>>>> >>>>>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF >>>>>> inet addr:10.100.100.2 Bcast:10.100.100.255 >>>> Mask:255.255.255.0 >>>>>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link >>>>>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 >>>>>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 >> frame:0 >>>>>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 >>>>>> collisions:0 txqueuelen:1000 >>>>>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB) >>>>>> >>>>>> Each interface has the MTU set to 9000, the driver is the one provided >>>>>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled and >>>>>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too). >>>>>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no >> success. >>>>>> Single CPU load is no more than 10% with 8000 interrupt/s: >>>>>> >>>>>> [r...@grids2 ~]# mpstat -P ALL 2 >>>>>> >>>>>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft >> %steal >>>>>> %idle intr/s >>>>>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 >> 0.00 >>>>>> 98.69 9018.00 >>>>>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1002.50 >>>>>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1.50 >>>>>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 >> 0.00 >>>>>> 89.55 8007.00 >>>>>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1.00 >>>>>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1.00 >>>>>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 3.00 >>>>>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1.00 >>>>>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 >>>>>> 100.00 1.00 >>>>>> >>>>>> I tried to configure smp affinity to assign the IRQ of each >>>>>> rx-queue to a different CPU, with no effect on the amount of dropped >>>>>> packets. I also verified that if I connect the NIC on a 10GE switch >> and >>>>>> send more than one stream from different servers, the card uses >>>>>> different rx-queues but the problem still exists. >>>>>> >>>>>> I modified the /etc/sysctl.conf file parameters adding: >>>>>> >>>>>> net.core.wmem_max = 16777216 >>>>>> net.core.wmem_default = 8388608 >>>>>> net.core.rmem_max = 16777216 >>>>>> net.core.rmem_default = 8388608 >>>>>> net.ipv4.tcp_rmem = 4096 262144 16777216 >>>>>> net.ipv4.tcp_wmem = 4096 262144 16777216 >>>>>> net.ipv4.tcp_mem = 4096 8388608 16777216 >>>>>> net.core.optmem_max = 524288 >>>>>> net.core.netdev_max_backlog = 200000 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Some other details: >>>>>> >>>>>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading >> disabled) >>>>>> - RAM: 24 GB (12 X 2GB) >>>>>> - Slot PCI: PCIe gen.2 x8 >>>>>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5 >>>>>> >>>>>> Any suggestions? >>>>>> Thank you very much in advance. >>>>>> >>>>>> Mirko Corosu >>>>>> >>>>>> -- >>>>>> >>>>>> ---------------------------------------------------------------------- >> -- >>>> -- >>>>>> Mirko Corosu >>>>>> Network and system administrator >>>>>> Computing Center >>>>>> Istituto Nazionale Fisica Nucleare >>>>>> Via Dodecaneso 33 >>>>>> 16146 Genova, Italy >>>>>> http://www.ge.infn.it >>>>>> ---------------------------------------------------------------------- >> -- >>>> -- >>>>>> >>>>>> ---------------------------------------------------------------------- >> -- >>>> --- >>>>>> --- >>>>>> Download Intel® Parallel Studio Eval >>>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>>> proactively, and fine-tune applications for parallel performance. >>>>>> See why Intel Parallel Studio got high marks during beta. >>>>>> http://p.sf.net/sfu/intel-sw-dev >>>>>> _______________________________________________ >>>>>> E1000-devel mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>>>> To learn more about Intel® Ethernet, visit >>>>>> http://communities.intel.com/community/wired >>>> -- >>>> >>>> ------------------------------------------------------------------------ >> -- >>>> Mirko Corosu >>>> Istituto Nazionale Fisica Nucleare >>>> Via Dodecaneso 33 >>>> 16146 Genova >>>> www.ge.infn.it >>>> Phone +39 010 3536361 >>>> ------------------------------------------------------------------------ >> -- >>>> ------------------------------------------------------------------------ >> -- >>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >> sbagliata. >>>> ------------------------------------------------------------------------ >> -- >> >> -- >> >> -------------------------------------------------------------------------- >> Mirko Corosu >> Network and system administrator >> Computing Center >> Istituto Nazionale Fisica Nucleare >> Via Dodecaneso 33 >> 16146 Genova, Italy >> http://www.ge.infn.it >> -------------------------------------------------------------------------- >> >> >> >> -------------------------------------------------------------------------- >> Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >> -------------------------------------------------------------------------- > -- -------------------------------------------------------------------------- Mirko Corosu Network and system administrator Computing Center Istituto Nazionale Fisica Nucleare Via Dodecaneso 33 16146 Genova, Italy http://www.ge.infn.it -------------------------------------------------------------------------- -------------------------------------------------------------------------- Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. -------------------------------------------------------------------------- ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
