Related to reducing the likelihood of a backlog of transmits at 1G, try Disable TSO - ethtool -K eth0 tso off Reduce TX descriptors - ethtool -G eth0 tx 64
I would try the first by itself, then try the two combined. -- Jesse Brandeburg > On Feb 4, 2021, at 7:55 AM, Fujinaka, Todd <todd.fujin...@intel.com> wrote: > > I checked with our performance team and I think the only thing we can see is > the possible bottleneck with your link partner. We don't do any > interoperability testing with Netgear, and are unaware if they have any > equipment that isn't consumer-grade. > > I would suggest that you follow up with Ubuntu. They will contact us if they > need any further help with the issue. > > Todd Fujinaka > Software Application Engineer > Data Center Group > Intel Corporation > todd.fujin...@intel.com > > -----Original Message----- > From: Pekka Pietikäinen <pe...@pietikainen.org> > Sent: Thursday, February 4, 2021 2:15 AM > To: e1000-devel@lists.sourceforge.net > Subject: [E1000-devel] Fake Tx hangs with ixgbe 5.10.2 > > Hi, > > We were seeing a lot of Tx hangs (once a day in lab, much more in > production) with Ubuntu 18.04.5 built-in driver (5.0.0-k and 5.1.0-k), > causing link resets etc.. Trying to isolate the problem a bit more we're now > trying the latest out-of-tree driver. Now it's > > Feb 03 22:46:19 x kernel: NETDEV WATCHDOG: enp13s0 (ixgbe): transmit queue 3 > timed out Feb 03 22:46:19 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang > detected with timeout of 5 seconds Feb 04 01:56:01 x kernel: ixgbe > 0000:0d:00.0 enp13s0: Fake Tx hang detected with timeout of 10 seconds Feb 04 > 15:13:21 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang detected with > timeout of 20 seconds > > "Fake Tx Hang", but traffic still stops for quite a while. Setup is > > ixgbe 0000:0d:00.0: enabling device (0000 -> 0002) ixgbe 0000:0d:00.0 > 0000:0d:00.0 (uninitialized): ixgbe_check_options: > FCoE Offload feature enabled > ixgbe 0000:0d:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = > 8 XDP Queue count = 0 ixgbe 0000:0d:00.0: 32.000 Gb/s available PCIe > bandwidth, limited by 5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s > with 8 GT/s x8 link) ixgbe 0000:0d:00.0 eth0: MAC: 2, PHY: 20, SFP+: 5, PBA > No: E68787-011 > > ixgbe 0000:0d:00.0 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash ixgbe > 0000:0d:00.0 eth0: Intel(R) 10 Gigabit Network Connection > > talking via 10Gbps fibre to Netgear switch to 20 clients (1GbE over copper) > > Disabling irqbalance and using set_irq_affinity (-x local) results in > > /proc/interrupts > > 132: 6932 1128 1977 2921 7857 7960 > 4392 3754 IR-PCI-MSI 520192-edge enp0s31f6 > 133: 264128 280513 208862 114758 608511 > 839 90376 187222 IR-PCI-MSI 6815744-edge enp13s0-TxRx-0 > 134: 2844471 365014 105711 1009702 423599 > 688773 0 589578 IR-PCI-MSI 6815745-edge enp13s0-TxRx-1 > 135: 95567 140332 43052 46623 86704 57976 > 135573 167195 IR-PCI-MSI 6815746-edge enp13s0-TxRx-2 > 136: 318887 197761 625025 193550 101415 > 102114 55755 291272 IR-PCI-MSI 6815747-edge enp13s0-TxRx-3 > 137: 491537 214000 158900 244269 266167 97220 > 25460 289919 IR-PCI-MSI 6815748-edge enp13s0-TxRx-4 > 138: 65767 248812 115232 238502 80103 46189 > 50782 81833 IR-PCI-MSI 6815749-edge enp13s0-TxRx-5 > 139: 207157 260936 48883 47421 216735 97253 > 92818 89079 IR-PCI-MSI 6815750-edge enp13s0-TxRx-6 > 140: 21512 424646 5111 390019 441436 4728 > 291370 278387 IR-PCI-MSI 6815751-edge enp13s0-TxRx-7 > 141: 0 0 0 3 0 0 > 0 0 IR-PCI-MSI 6815752-edge enp13s0 > > TxRx-1 on CPU0 seems a bit unbalanced, but otherwise fine? > > ethtool would suggest it's not flow control related (which would explain > things?), now experimenting with tuning interrupt moderation / ring size / > avoiding CPU0 . Is there anything else to try? > > NIC statistics: > rx_packets: 188092169 > tx_packets: 59581798 > rx_bytes: 263463989424 > tx_bytes: 25727566814 > rx_errors: 0 > tx_errors: 0 > rx_dropped: 0 > tx_dropped: 0 > multicast: 1237 > collisions: 0 > rx_over_errors: 0 > rx_crc_errors: 0 > rx_frame_errors: 0 > rx_fifo_errors: 0 > rx_missed_errors: 0 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > rx_pkts_nic: 188092169 > tx_pkts_nic: 59581798 > rx_bytes_nic: 264216358100 > tx_bytes_nic: 25965914840 > lsc_int: 3 > tx_busy: 0 > non_eop_descs: 0 > broadcast: 259 > rx_no_buffer_count: 0 > tx_timeout_count: 0 > tx_restart_queue: 4 > rx_length_errors: 0 > rx_long_length_errors: 0 > rx_short_length_errors: 0 > tx_flow_control_xon: 0 > rx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > rx_flow_control_xoff: 0 > rx_csum_offload_errors: 0 > alloc_rx_page: 40526166 > alloc_rx_page_failed: 0 > alloc_rx_buff_failed: 0 > rx_no_dma_resources: 0 > hw_rsc_aggregated: 0 > hw_rsc_flushed: 0 > fdir_match: 188027415 > fdir_miss: 178920 > fdir_overflow: 0 > fcoe_bad_fccrc: 0 > fcoe_last_errors: 0 > rx_fcoe_dropped: 0 > rx_fcoe_packets: 0 > rx_fcoe_dwords: 0 > fcoe_noddp: 0 > fcoe_noddp_ext_buff: 0 > tx_fcoe_packets: 0 > tx_fcoe_dwords: 0 > os2bmc_rx_by_bmc: 0 > os2bmc_tx_by_bmc: 0 > os2bmc_tx_by_host: 0 > os2bmc_rx_by_host: 0 > tx_hwtstamp_timeouts: 0 > tx_hwtstamp_skipped: 0 > rx_hwtstamp_cleared: 0 > tx_queue_0_packets: 2551919 > tx_queue_0_bytes: 303310258 > tx_queue_1_packets: 34799436 > tx_queue_1_bytes: 2365301599 > tx_queue_2_packets: 726053 > tx_queue_2_bytes: 101796295 > tx_queue_3_packets: 5109959 > tx_queue_3_bytes: 4885506128 > tx_queue_4_packets: 4655587 > tx_queue_4_bytes: 4755028677 > tx_queue_5_packets: 2533314 > tx_queue_5_bytes: 2516349075 > tx_queue_6_packets: 4109721 > tx_queue_6_bytes: 5511688758 > tx_queue_7_packets: 5095809 > tx_queue_7_bytes: 5288586024 > tx_queue_8_packets: 0 > tx_queue_8_bytes: 0 > tx_queue_9_packets: 0 > tx_queue_9_bytes: 0 > tx_queue_10_packets: 0 > tx_queue_10_bytes: 0 > tx_queue_11_packets: 0 > tx_queue_11_bytes: 0 > tx_queue_12_packets: 0 > tx_queue_12_bytes: 0 > tx_queue_13_packets: 0 > tx_queue_13_bytes: 0 > tx_queue_14_packets: 0 > tx_queue_14_bytes: 0 > tx_queue_15_packets: 0 > tx_queue_15_bytes: 0 > tx_queue_16_packets: 0 > tx_queue_16_bytes: 0 > tx_queue_17_packets: 0 > tx_queue_17_bytes: 0 > tx_queue_18_packets: 0 > tx_queue_18_bytes: 0 > tx_queue_19_packets: 0 > tx_queue_19_bytes: 0 > tx_queue_20_packets: 0 > tx_queue_20_bytes: 0 > tx_queue_21_packets: 0 > tx_queue_21_bytes: 0 > tx_queue_22_packets: 0 > tx_queue_22_bytes: 0 > tx_queue_23_packets: 0 > tx_queue_23_bytes: 0 > tx_queue_24_packets: 0 > tx_queue_24_bytes: 0 > tx_queue_25_packets: 0 > tx_queue_25_bytes: 0 > tx_queue_26_packets: 0 > tx_queue_26_bytes: 0 > tx_queue_27_packets: 0 > tx_queue_27_bytes: 0 > tx_queue_28_packets: 0 > tx_queue_28_bytes: 0 > tx_queue_29_packets: 0 > tx_queue_29_bytes: 0 > tx_queue_30_packets: 0 > tx_queue_30_bytes: 0 > tx_queue_31_packets: 0 > tx_queue_31_bytes: 0 > tx_queue_32_packets: 0 > tx_queue_32_bytes: 0 > tx_queue_33_packets: 0 > tx_queue_33_bytes: 0 > tx_queue_34_packets: 0 > tx_queue_34_bytes: 0 > tx_queue_35_packets: 0 > tx_queue_35_bytes: 0 > tx_queue_36_packets: 0 > tx_queue_36_bytes: 0 > tx_queue_37_packets: 0 > tx_queue_37_bytes: 0 > tx_queue_38_packets: 0 > tx_queue_38_bytes: 0 > tx_queue_39_packets: 0 > tx_queue_39_bytes: 0 > tx_queue_40_packets: 0 > tx_queue_40_bytes: 0 > tx_queue_41_packets: 0 > tx_queue_41_bytes: 0 > tx_queue_42_packets: 0 > tx_queue_42_bytes: 0 > tx_queue_43_packets: 0 > tx_queue_43_bytes: 0 > tx_queue_44_packets: 0 > tx_queue_44_bytes: 0 > tx_queue_45_packets: 0 > tx_queue_45_bytes: 0 > tx_queue_46_packets: 0 > tx_queue_46_bytes: 0 > tx_queue_47_packets: 0 > tx_queue_47_bytes: 0 > tx_queue_48_packets: 0 > tx_queue_48_bytes: 0 > tx_queue_49_packets: 0 > tx_queue_49_bytes: 0 > tx_queue_50_packets: 0 > tx_queue_50_bytes: 0 > tx_queue_51_packets: 0 > tx_queue_51_bytes: 0 > tx_queue_52_packets: 0 > tx_queue_52_bytes: 0 > tx_queue_53_packets: 0 > tx_queue_53_bytes: 0 > tx_queue_54_packets: 0 > tx_queue_54_bytes: 0 > tx_queue_55_packets: 0 > tx_queue_55_bytes: 0 > tx_queue_56_packets: 0 > tx_queue_56_bytes: 0 > tx_queue_57_packets: 0 > tx_queue_57_bytes: 0 > tx_queue_58_packets: 0 > tx_queue_58_bytes: 0 > tx_queue_59_packets: 0 > tx_queue_59_bytes: 0 > tx_queue_60_packets: 0 > tx_queue_60_bytes: 0 > tx_queue_61_packets: 0 > tx_queue_61_bytes: 0 > tx_queue_62_packets: 0 > tx_queue_62_bytes: 0 > tx_queue_63_packets: 0 > tx_queue_63_bytes: 0 > tx_queue_64_packets: 0 > tx_queue_64_bytes: 0 > tx_queue_65_packets: 0 > tx_queue_65_bytes: 0 > tx_queue_66_packets: 0 > tx_queue_66_bytes: 0 > tx_queue_67_packets: 0 > tx_queue_67_bytes: 0 > tx_queue_68_packets: 0 > tx_queue_68_bytes: 0 > tx_queue_69_packets: 0 > tx_queue_69_bytes: 0 > tx_queue_70_packets: 0 > tx_queue_70_bytes: 0 > rx_queue_0_packets: 11217199 > rx_queue_0_bytes: 16124868046 > rx_queue_1_packets: 157176240 > rx_queue_1_bytes: 227080033991 > rx_queue_2_packets: 1501661 > rx_queue_2_bytes: 1972184904 > rx_queue_3_packets: 5075867 > rx_queue_3_bytes: 5607058971 > rx_queue_4_packets: 5326935 > rx_queue_4_bytes: 6020504920 > rx_queue_5_packets: 1606940 > rx_queue_5_bytes: 1282193306 > rx_queue_6_packets: 2148621 > rx_queue_6_bytes: 1388353208 > rx_queue_7_packets: 4038706 > rx_queue_7_bytes: 3988792078 > rx_queue_8_packets: 0 > rx_queue_8_bytes: 0 > rx_queue_9_packets: 0 > rx_queue_9_bytes: 0 > rx_queue_10_packets: 0 > rx_queue_10_bytes: 0 > rx_queue_11_packets: 0 > rx_queue_11_bytes: 0 > rx_queue_12_packets: 0 > rx_queue_12_bytes: 0 > rx_queue_13_packets: 0 > rx_queue_13_bytes: 0 > rx_queue_14_packets: 0 > rx_queue_14_bytes: 0 > rx_queue_15_packets: 0 > rx_queue_15_bytes: 0 > rx_queue_16_packets: 0 > rx_queue_16_bytes: 0 > rx_queue_17_packets: 0 > rx_queue_17_bytes: 0 > rx_queue_18_packets: 0 > rx_queue_18_bytes: 0 > rx_queue_19_packets: 0 > rx_queue_19_bytes: 0 > rx_queue_20_packets: 0 > rx_queue_20_bytes: 0 > rx_queue_21_packets: 0 > rx_queue_21_bytes: 0 > rx_queue_22_packets: 0 > rx_queue_22_bytes: 0 > rx_queue_23_packets: 0 > rx_queue_23_bytes: 0 > rx_queue_24_packets: 0 > rx_queue_24_bytes: 0 > rx_queue_25_packets: 0 > rx_queue_25_bytes: 0 > rx_queue_26_packets: 0 > rx_queue_26_bytes: 0 > rx_queue_27_packets: 0 > rx_queue_27_bytes: 0 > rx_queue_28_packets: 0 > rx_queue_28_bytes: 0 > rx_queue_29_packets: 0 > rx_queue_29_bytes: 0 > rx_queue_30_packets: 0 > rx_queue_30_bytes: 0 > rx_queue_31_packets: 0 > rx_queue_31_bytes: 0 > rx_queue_32_packets: 0 > rx_queue_32_bytes: 0 > rx_queue_33_packets: 0 > rx_queue_33_bytes: 0 > rx_queue_34_packets: 0 > rx_queue_34_bytes: 0 > rx_queue_35_packets: 0 > rx_queue_35_bytes: 0 > rx_queue_36_packets: 0 > rx_queue_36_bytes: 0 > rx_queue_37_packets: 0 > rx_queue_37_bytes: 0 > rx_queue_38_packets: 0 > rx_queue_38_bytes: 0 > rx_queue_39_packets: 0 > rx_queue_39_bytes: 0 > rx_queue_40_packets: 0 > rx_queue_40_bytes: 0 > rx_queue_41_packets: 0 > rx_queue_41_bytes: 0 > rx_queue_42_packets: 0 > rx_queue_42_bytes: 0 > rx_queue_43_packets: 0 > rx_queue_43_bytes: 0 > rx_queue_44_packets: 0 > rx_queue_44_bytes: 0 > rx_queue_45_packets: 0 > rx_queue_45_bytes: 0 > rx_queue_46_packets: 0 > rx_queue_46_bytes: 0 > rx_queue_47_packets: 0 > rx_queue_47_bytes: 0 > rx_queue_48_packets: 0 > rx_queue_48_bytes: 0 > rx_queue_49_packets: 0 > rx_queue_49_bytes: 0 > rx_queue_50_packets: 0 > rx_queue_50_bytes: 0 > rx_queue_51_packets: 0 > rx_queue_51_bytes: 0 > rx_queue_52_packets: 0 > rx_queue_52_bytes: 0 > rx_queue_53_packets: 0 > rx_queue_53_bytes: 0 > rx_queue_54_packets: 0 > rx_queue_54_bytes: 0 > rx_queue_55_packets: 0 > rx_queue_55_bytes: 0 > rx_queue_56_packets: 0 > rx_queue_56_bytes: 0 > rx_queue_57_packets: 0 > rx_queue_57_bytes: 0 > rx_queue_58_packets: 0 > rx_queue_58_bytes: 0 > rx_queue_59_packets: 0 > rx_queue_59_bytes: 0 > rx_queue_60_packets: 0 > rx_queue_60_bytes: 0 > rx_queue_61_packets: 0 > rx_queue_61_bytes: 0 > rx_queue_62_packets: 0 > rx_queue_62_bytes: 0 > rx_queue_63_packets: 0 > rx_queue_63_bytes: 0 > rx_queue_64_packets: 0 > rx_queue_64_bytes: 0 > rx_queue_65_packets: 0 > rx_queue_65_bytes: 0 > rx_queue_66_packets: 0 > rx_queue_66_bytes: 0 > rx_queue_67_packets: 0 > rx_queue_67_bytes: 0 > rx_queue_68_packets: 0 > rx_queue_68_bytes: 0 > rx_queue_69_packets: 0 > rx_queue_69_bytes: 0 > rx_queue_70_packets: 0 > rx_queue_70_bytes: 0 > tx_pb_0_pxon: 0 > tx_pb_0_pxoff: 0 > tx_pb_1_pxon: 0 > tx_pb_1_pxoff: 0 > tx_pb_2_pxon: 0 > tx_pb_2_pxoff: 0 > tx_pb_3_pxon: 0 > tx_pb_3_pxoff: 0 > tx_pb_4_pxon: 0 > tx_pb_4_pxoff: 0 > tx_pb_5_pxon: 0 > tx_pb_5_pxoff: 0 > tx_pb_6_pxon: 0 > tx_pb_6_pxoff: 0 > tx_pb_7_pxon: 0 > tx_pb_7_pxoff: 0 > rx_pb_0_pxon: 0 > rx_pb_0_pxoff: 0 > rx_pb_1_pxon: 0 > rx_pb_1_pxoff: 0 > rx_pb_2_pxon: 0 > rx_pb_2_pxoff: 0 > rx_pb_3_pxon: 0 > rx_pb_3_pxoff: 0 > rx_pb_4_pxon: 0 > rx_pb_4_pxoff: 0 > rx_pb_5_pxon: 0 > rx_pb_5_pxoff: 0 > rx_pb_6_pxon: 0 > rx_pb_6_pxoff: 0 > rx_pb_7_pxon: 0 > rx_pb_7_pxoff: 0 > > > > > > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
