Sorry for the late, I missed the last message. > Mirko, did you ever try sending two UDP streams? >
Yes, I tried also more than one UDP stream with no luck. > > rx_missed_error is the number of packets that have been dropped due to > no HOST memory buffers available. The driver provides host memory to > the controller via the driver's interrupt routine. > > In your case due to your single threaded workload you're limited by what > a single cpu can process. The output of mpstat shows very low cpu load during the tests. I contacted Dell support to investigate any possible bottleneck due to the internal server architecture. > > On Thu, 2010-02-25 at 00:56 -0800, Mirko Corosu wrote: >> Hi Donald, >> >>> Once again to clarify your concern is: >>> >>> TX system -> RX system (TCP) 9.5 Gb/s >>> TX system -> RX system (UDP / FC off) 5.6 Gb/s >> >> Sorry... not exactly, but I guess I found the reason why we can't >> understand each other. When I say: "on TX side I get 9.5 Gb/s" I mean >> that I measure the network traffic through the interface on the TX >> system (I use dstat for that). >> >> I try to clarify further my numbers: >> >> All tests are from TX system to RX system >> >> UDP (no FC): >> through eth0 on TX system: 9.5 Gb/s >> through eth0 on RX system: 5.6 Gb/s >> >> UDP (FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> TCP (no FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> TCP (FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> FC does not improve my throughput (but of course eliminates >> rx_missed_errors). >> >>> As for the meaning of the rx_missed_errors I can at least help you with >>> that. I imagined you looked at the code but it's the summation of an array >>> of registers called MPC(), one for each receive FIFO. It's incremented >>> when Packets are missed due to insufficient space to store that packet. >>> This might be due to bandwidth issue with the bus IO (which is why I was >>> asking about the slot you have the NIC's in) or too few buffers allocated. >>> If I remember correctly you already tried that. Or in our case a >>> transmitter blasting out packets as faster than we can receive them >> Ok. As I wrote before this is my network buffer configuration: >> >> net.core.wmem_max = 16777216 >> net.core.wmem_default = 8388608 >> net.core.rmem_max = 16777216 >> net.core.rmem_default = 8388608 >> net.ipv4.tcp_rmem = 4096 262144 16777216 >> net.ipv4.tcp_wmem = 4096 262144 16777216 >> net.ipv4.tcp_mem = 4096 8388608 16777216 >> net.core.optmem_max = 524288 >> net.core.netdev_max_backlog = 200000 >> >> AFAIK, these are the only parameters I can set. >> >> Thanks for your patience :-) >> >> Mirko >> >>>> -----Original Message----- >>>> From: Mirko Corosu [mailto:[email protected]] >>>> Sent: Wednesday, February 24, 2010 2:56 AM >>>> To: [email protected] >>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>> >>>> Hi Donald, >>>> >>>> >>>>> One thing I did notice was that the Receive side has 8 rx queues while >>>> the Transmit has 16 rx queues. You mentioned these were identical machines >>>> could Hyper-Threading be off on the Receiving machine? >>>> Yes, it was my fault. I forgot to disable hyperthreading on TX server. >>>> Now it is disabled, but the problem still remains. >>>> >>>>> Also I'm still a little confused about how your test is set up. If you >>>> have four netperf's running on, as you called it, your Tx machine and your >>>> Rx is only running netserver then I would only expect heavy traffic from Tx >>>> -> Rx. Or am I missing some thing here? >>>> I try to summarize my tests. >>>> >>>> For now I am testing only mono-directional TCP and UDP stream (from TX >>>> server to RX server directly ). What I cannot understand is the low >>>> throughput that I'm getting. The TCP tests shows a throughput of only >>>> 5.6 Gb/s. During UDP tests, with flow control disabled, I measured on TX >>>> side a throughput of about 9.5 Gb/s (so the card can send at wire speed) >>>> but on RX side I see only 5.6 Gb/s. I attribute this problem to the high >>>> number of rx_missed_errors but I cannot figure out what is the real >>>> cause (I don't know even what rx_missed_errors actually mean....). >>>> >>>> Thanks >>>> >>>> Mirko >>>> >>>>>> -----Original Message----- >>>>>> From: Mirko Corosu [mailto:[email protected]] >>>>>> Sent: Tuesday, February 23, 2010 3:30 AM >>>>>> To: Skidmore, Donald C >>>>>> Cc: [email protected] >>>>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>>> >>>>>> Hi Donald, >>>>>> >>>>>> Ok, sorry for the lack of informations: I'm not used to interact with >>>>>> developers. :-) >>>>>> >>>>>> >>>>>>> How were you using netperf in your tests? What tests (I assume >>>>>> [UDP|TCP]_STREAM) and how many instances of netperf were running each >>>> side? >>>>>> I'm running four instances of UDP_STREAM or TCP_STREAM tests on the tx >>>>>> side, with different packet sizes (from 1000 to 9000 byte) and a single >>>>>> netserver on the rx side. >>>>>> >>>>>>> From the spec I got off the web on the R710 I can see they have either >>>> (2 >>>>>> PCIe x8 and 2 PCIe x4) or (1 PCIe x16 and 2 PCIe x4 ). I was just >>>>>> concerned you might be plugged into one of the x4 slots. >>>>>> >>>>>> I have 2 PCIe x8 on the so called "Riser 2" and 2 PCIe x4 on the "Riser >>>>>> 1". On each server the Intel cards are plugged into the Riser 2 slots. >>>>>> >>>>>>>>> - Did you look at the dmesg for anything relative? >>>>>>>> I see no error or warning messages on both side. >>>>>>> It's not just error or warning messages I was interested in, although >>>>>> they would be interesting too. :) We also printk a message that >>>> displays >>>>>> the type of PCIe link we have. This would answer the question I posed >>>>>> above about what slot the cards were in. >>>>>> On rx side: >>>>>> >>>>>> [r...@grids2 ~]# dmesg |grep ixgbe >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> >>>>>> >>>>>> >>>>>> On tx side: >>>>>> >>>>>> [r...@client20 ~]# dmesg |grep ixgbe >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> >>>>>> >>>>>> >>>>>>> All that said we would expect to see rx_missed_errors with flow control >>>>>> disabled. Since nothing would be stopping packets from overrunning the >>>>>> buffer. Your test seems to be agreeing with this as the number of >>>>>> rx_missed_errors goes down with TCP (like you said) most likely due to >>>> TCP >>>>>> sliding window. >>>>>>> You don't see this issue with flow control enabled, right? >>>>>> Right, but the throughput remains about 5.6 Gb/s and a can't figure out >>>>>> where the bottleneck could be.... >>>>>> >>>>>> Thanks a lot >>>>>> >>>>>> Mirko >>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mirko Corosu [mailto:[email protected]] >>>>>>>> Sent: Saturday, February 20, 2010 10:50 AM >>>>>>>> To: Skidmore, Donald C >>>>>>>> Cc: [email protected] >>>>>>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx >>>> packet >>>>>>>> Hi Donald, >>>>>>>> >>>>>>>> Thank you for your reply. >>>>>>>> >>>>>>>>> I have a couple additional questions >>>>>>>>> - What were you running to do this test? (i.e. netperf, pktgen?) >>>>>>>> netperf-2.4.2-1.el5.rf >>>>>>>> >>>>>>>>> - What throughput were you seeing? >>>>>>>> On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s >>>>>>>> >>>>>>>>> - Are both ports plugged into x8 PCIe slots >>>>>>>> Yes, both servers are Dell R710 with the same hardware configuration >>>>>>>> >>>>>>>>> - Did you look at the dmesg for anything relative? >>>>>>>> I see no error or warning messages on both side. >>>>>>>> >>>>>>>>> - Do you see this issue only with UDP? >>>>>>>> If I send a TCP stream I see on both side a throughput of about 5.54 >>>>>>>> Gb/s and less rx_missed_errors on rx side: >>>>>>>> >>>>>>>> NIC statistics: >>>>>>>> rx_packets: 1768101 >>>>>>>> tx_packets: 235384 >>>>>>>> rx_bytes: 15937257320 >>>>>>>> tx_bytes: 15704254 >>>>>>>> lsc_int: 2 >>>>>>>> tx_busy: 0 >>>>>>>> non_eop_descs: 7072222 >>>>>>>> rx_errors: 0 >>>>>>>> tx_errors: 0 >>>>>>>> rx_dropped: 0 >>>>>>>> tx_dropped: 0 >>>>>>>> multicast: 1 >>>>>>>> broadcast: 0 >>>>>>>> rx_no_buffer_count: 0 >>>>>>>> collisions: 0 >>>>>>>> rx_over_errors: 0 >>>>>>>> rx_crc_errors: 0 >>>>>>>> rx_frame_errors: 0 >>>>>>>> rx_fifo_errors: 0 >>>>>>>> rx_missed_errors: 1539 >>>>>>>> tx_aborted_errors: 0 >>>>>>>> tx_carrier_errors: 0 >>>>>>>> tx_fifo_errors: 0 >>>>>>>> tx_heartbeat_errors: 0 >>>>>>>> tx_timeout_count: 0 >>>>>>>> tx_restart_queue: 0 >>>>>>>> rx_long_length_errors: 0 >>>>>>>> rx_short_length_errors: 0 >>>>>>>> tx_tcp4_seg_ctxt: 0 >>>>>>>> tx_tcp6_seg_ctxt: 0 >>>>>>>> tx_flow_control_xon: 0 >>>>>>>> rx_flow_control_xon: 0 >>>>>>>> tx_flow_control_xoff: 0 >>>>>>>> rx_flow_control_xoff: 0 >>>>>>>> rx_csum_offload_good: 1768098 >>>>>>>> rx_csum_offload_errors: 0 >>>>>>>> tx_csum_offload_ctxt: 11 >>>>>>>> low_latency_interrupt: 0 >>>>>>>> alloc_rx_page_failed: 0 >>>>>>>> alloc_rx_buff_failed: 0 >>>>>>>> lro_aggregated: 1760417 >>>>>>>> lro_flushed: 375277 >>>>>>>> lro_recycled: 1012791 >>>>>>>> rx_no_dma_resources: 0 >>>>>>>> hw_rsc_count: 0 >>>>>>>> rx_flm: 0 >>>>>>>> fdir_match: 0 >>>>>>>> fdir_miss: 0 >>>>>>>> tx_queue_0_packets: 235384 >>>>>>>> tx_queue_0_bytes: 15704254 >>>>>>>> rx_queue_0_packets: 2 >>>>>>>> rx_queue_0_bytes: 120 >>>>>>>> rx_queue_1_packets: 8 >>>>>>>> rx_queue_1_bytes: 786 >>>>>>>> rx_queue_2_packets: 301752 >>>>>>>> rx_queue_2_bytes: 2719959436 >>>>>>>> rx_queue_3_packets: 562919 >>>>>>>> rx_queue_3_bytes: 5074118826 >>>>>>>> rx_queue_4_packets: 7 >>>>>>>> rx_queue_4_bytes: 726 >>>>>>>> rx_queue_5_packets: 15 >>>>>>>> rx_queue_5_bytes: 1526 >>>>>>>> rx_queue_6_packets: 313901 >>>>>>>> rx_queue_6_bytes: 2829476778 >>>>>>>> rx_queue_7_packets: 589497 >>>>>>>> rx_queue_7_bytes: 5313699122 >>>>>>>> >>>>>>>> I guess it's because of TCP flow control. >>>>>>>> >>>>>>>> I remember that the same behavior was triggered by enabling flow >>>> control >>>>>>>> on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to >>>>>>>> wait till Monday to perform another run to test that configuration. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Mirko >>>>>>>> >>>>>>>>> What I'm wondering is if this NIC was receiving faster than it could >>>>>>>> process them. >>>>>>>>> Thanks, >>>>>>>>> -Don Skidmore <[email protected]> >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mirko Corosu [mailto:[email protected]] >>>>>>>>>> Sent: Friday, February 19, 2010 5:30 AM >>>>>>>>>> To: [email protected] >>>>>>>>>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>>>>>>> >>>>>>>>>> Dear all, >>>>>>>>>> >>>>>>>>>> I'm new to this list, if this isn't the right place to post this >>>>>> issue, >>>>>>>>>> please let me know. >>>>>>>>>> >>>>>>>>>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on >>>>>> two >>>>>>>>>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The >>>> two >>>>>>>>>> NICs are directly connected each other (no switch in between). I >>>> am >>>>>>>>>> experiencing a problem on the receive side of the connection: I'm >>>>>>>> sending >>>>>>>>>> an UDP stream after having disabled flow control on each card, and >>>>>> about >>>>>>>>>> the 60% of the transmitted packets are dropped with a large number >>>> of >>>>>>>>>> rx_missed_errors. This is what ifconfig and ethtool -S show: >>>>>>>>>> >>>>>>>>>> [r...@grids2]# ethtool -S eth0 >>>>>>>>>> NIC statistics: >>>>>>>>>> rx_packets: 1761794 >>>>>>>>>> tx_packets: 32 >>>>>>>>>> rx_bytes: 14748667944 >>>>>>>>>> tx_bytes: 5746 >>>>>>>>>> lsc_int: 2 >>>>>>>>>> tx_busy: 0 >>>>>>>>>> non_eop_descs: 6510828 >>>>>>>>>> rx_errors: 0 >>>>>>>>>> tx_errors: 0 >>>>>>>>>> rx_dropped: 0 >>>>>>>>>> tx_dropped: 0 >>>>>>>>>> multicast: 2 >>>>>>>>>> broadcast: 2 >>>>>>>>>> rx_no_buffer_count: 0 >>>>>>>>>> collisions: 0 >>>>>>>>>> rx_over_errors: 0 >>>>>>>>>> rx_crc_errors: 0 >>>>>>>>>> rx_frame_errors: 0 >>>>>>>>>> rx_fifo_errors: 0 >>>>>>>>>> rx_missed_errors: 1148382 >>>>>>>>>> tx_aborted_errors: 0 >>>>>>>>>> tx_carrier_errors: 0 >>>>>>>>>> tx_fifo_errors: 0 >>>>>>>>>> tx_heartbeat_errors: 0 >>>>>>>>>> tx_timeout_count: 0 >>>>>>>>>> tx_restart_queue: 0 >>>>>>>>>> rx_long_length_errors: 0 >>>>>>>>>> rx_short_length_errors: 0 >>>>>>>>>> tx_tcp4_seg_ctxt: 0 >>>>>>>>>> tx_tcp6_seg_ctxt: 0 >>>>>>>>>> tx_flow_control_xon: 0 >>>>>>>>>> rx_flow_control_xon: 0 >>>>>>>>>> tx_flow_control_xoff: 0 >>>>>>>>>> rx_flow_control_xoff: 0 >>>>>>>>>> rx_csum_offload_good: 28 >>>>>>>>>> rx_csum_offload_errors: 0 >>>>>>>>>> tx_csum_offload_ctxt: 11 >>>>>>>>>> low_latency_interrupt: 0 >>>>>>>>>> alloc_rx_page_failed: 0 >>>>>>>>>> alloc_rx_buff_failed: 0 >>>>>>>>>> lro_aggregated: 8 >>>>>>>>>> lro_flushed: 8 >>>>>>>>>> lro_recycled: 0 >>>>>>>>>> rx_no_dma_resources: 0 >>>>>>>>>> hw_rsc_count: 0 >>>>>>>>>> rx_flm: 0 >>>>>>>>>> fdir_match: 0 >>>>>>>>>> fdir_miss: 0 >>>>>>>>>> tx_queue_0_packets: 32 >>>>>>>>>> tx_queue_0_bytes: 5746 >>>>>>>>>> rx_queue_0_packets: 2 >>>>>>>>>> rx_queue_0_bytes: 120 >>>>>>>>>> rx_queue_1_packets: 2 >>>>>>>>>> rx_queue_1_bytes: 120 >>>>>>>>>> rx_queue_2_packets: 1761762 >>>>>>>>>> rx_queue_2_bytes: 14748664800 >>>>>>>>>> rx_queue_3_packets: 0 >>>>>>>>>> rx_queue_3_bytes: 0 >>>>>>>>>> rx_queue_4_packets: 0 >>>>>>>>>> rx_queue_4_bytes: 0 >>>>>>>>>> rx_queue_5_packets: 14 >>>>>>>>>> rx_queue_5_bytes: 1452 >>>>>>>>>> rx_queue_6_packets: 7 >>>>>>>>>> rx_queue_6_bytes: 726 >>>>>>>>>> rx_queue_7_packets: 7 >>>>>>>>>> rx_queue_7_bytes: 726 >>>>>>>>>> >>>>>>>>>> [r...@grids2]# ifconfig eth0 >>>>>>>>>> >>>>>>>>>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF >>>>>>>>>> inet addr:10.100.100.2 Bcast:10.100.100.255 >>>>>>>> Mask:255.255.255.0 >>>>>>>>>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link >>>>>>>>>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 >>>>>>>>>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 >>>>>> frame:0 >>>>>>>>>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>>>>> collisions:0 txqueuelen:1000 >>>>>>>>>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB) >>>>>>>>>> >>>>>>>>>> Each interface has the MTU set to 9000, the driver is the one >>>> provided >>>>>>>>>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled >>>> and >>>>>>>>>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too). >>>>>>>>>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no >>>>>> success. >>>>>>>>>> Single CPU load is no more than 10% with 8000 interrupt/s: >>>>>>>>>> >>>>>>>>>> [r...@grids2 ~]# mpstat -P ALL 2 >>>>>>>>>> >>>>>>>>>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft >>>>>> %steal >>>>>>>>>> %idle intr/s >>>>>>>>>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 >>>>>> 0.00 >>>>>>>>>> 98.69 9018.00 >>>>>>>>>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1002.50 >>>>>>>>>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.50 >>>>>>>>>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 >>>>>> 0.00 >>>>>>>>>> 89.55 8007.00 >>>>>>>>>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 3.00 >>>>>>>>>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> >>>>>>>>>> I tried to configure smp affinity to assign the IRQ of each >>>>>>>>>> rx-queue to a different CPU, with no effect on the amount of dropped >>>>>>>>>> packets. I also verified that if I connect the NIC on a 10GE switch >>>>>> and >>>>>>>>>> send more than one stream from different servers, the card uses >>>>>>>>>> different rx-queues but the problem still exists. >>>>>>>>>> >>>>>>>>>> I modified the /etc/sysctl.conf file parameters adding: >>>>>>>>>> >>>>>>>>>> net.core.wmem_max = 16777216 >>>>>>>>>> net.core.wmem_default = 8388608 >>>>>>>>>> net.core.rmem_max = 16777216 >>>>>>>>>> net.core.rmem_default = 8388608 >>>>>>>>>> net.ipv4.tcp_rmem = 4096 262144 16777216 >>>>>>>>>> net.ipv4.tcp_wmem = 4096 262144 16777216 >>>>>>>>>> net.ipv4.tcp_mem = 4096 8388608 16777216 >>>>>>>>>> net.core.optmem_max = 524288 >>>>>>>>>> net.core.netdev_max_backlog = 200000 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Some other details: >>>>>>>>>> >>>>>>>>>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading >>>>>> disabled) >>>>>>>>>> - RAM: 24 GB (12 X 2GB) >>>>>>>>>> - Slot PCI: PCIe gen.2 x8 >>>>>>>>>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5 >>>>>>>>>> >>>>>>>>>> Any suggestions? >>>>>>>>>> Thank you very much in advance. >>>>>>>>>> >>>>>>>>>> Mirko Corosu >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> -- >>>>>>>>>> Mirko Corosu >>>>>>>>>> Network and system administrator >>>>>>>>>> Computing Center >>>>>>>>>> Istituto Nazionale Fisica Nucleare >>>>>>>>>> Via Dodecaneso 33 >>>>>>>>>> 16146 Genova, Italy >>>>>>>>>> http://www.ge.infn.it >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> -- >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> --- >>>>>>>>>> --- >>>>>>>>>> Download Intel® Parallel Studio Eval >>>>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>>>>>>> proactively, and fine-tune applications for parallel performance. >>>>>>>>>> See why Intel Parallel Studio got high marks during beta. >>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev >>>>>>>>>> _______________________________________________ >>>>>>>>>> E1000-devel mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>>>>>>>> To learn more about Intel® Ethernet, visit >>>>>>>>>> http://communities.intel.com/community/wired >>>>>>>> -- >>>>>>>> >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> Mirko Corosu >>>>>>>> Istituto Nazionale Fisica Nucleare >>>>>>>> Via Dodecaneso 33 >>>>>>>> 16146 Genova >>>>>>>> www.ge.infn.it >>>>>>>> Phone +39 010 3536361 >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >>>>>> sbagliata. >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>> >>>>>> -- >>>>>> >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> Mirko Corosu >>>>>> Network and system administrator >>>>>> Computing Center >>>>>> Istituto Nazionale Fisica Nucleare >>>>>> Via Dodecaneso 33 >>>>>> 16146 Genova, Italy >>>>>> http://www.ge.infn.it >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >>>> sbagliata. >>>>>> ------------------------------------------------------------------------ >>>> -- >>>> >>>> -- >>>> -------------------------------------------------------------------------- >>>> Mirko Corosu >>>> Network and system administrator >>>> Computing Center >>>> Istituto Nazionale Fisica Nucleare >>>> Via Dodecaneso 33 >>>> 16146 Genova, Italy >>>> http://www.ge.infn.it >>>> -------------------------------------------------------------------------- >>>> >>>> -------------------------------------------------------------------------- >>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> >>>> --------------------------------------------------------------------------- >>>> --- >>>> Download Intel® Parallel Studio Eval >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> _______________________________________________ >>>> E1000-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>> To learn more about Intel® Ethernet, visit >>>> http://communities.intel.com/community/wired >> >> -- >> -------------------------------------------------------------------------- >> Mirko Corosu >> Network and system administrator >> Computing Center >> Istituto Nazionale Fisica Nucleare >> Via Dodecaneso 33 >> 16146 Genova, Italy >> http://www.ge.infn.it >> -------------------------------------------------------------------------- >> >> -------------------------------------------------------------------------- >> Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >> -------------------------------------------------------------------------- >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> E1000-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® Ethernet, visit >> http://communities.intel.com/community/wired > -- -------------------------------------------------------------------------- Mirko Corosu Network and system administrator Computing Center Istituto Nazionale Fisica Nucleare Via Dodecaneso 33 16146 Genova, Italy http://www.ge.infn.it -------------------------------------------------------------------------- ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
