On 12 Nov 2020, at 14:31, Jesper Dangaard Brouer wrote:
> On Thu, 12 Nov 2020 12:26:20 +0100 > "Thomas Rosenstein" <[email protected]> wrote: > >>> Long-shot theory(2): The NIC driver IRQ interrupt coalesce >>> adaptive-IRQ (adj via ethtool -C) is wrong. The reason it is wrong is >>> because of TSO frames, due packet-to-byte ratio isn't taken into >>> account. >> >> here are the coalese parameters: (are the same for 3.10 and 5.4.60) >> >> ethtool -c eth4 >> Coalesce parameters for eth4: >> Adaptive RX: on TX: on > > Notice "Adaptive" setting is on. My long-shot theory(2) is that this > adaptive algorithm in the driver code can guess wrong (due to not > taking TSO into account) and cause issues for > > Try to turn this adaptive algorithm off: > > ethtool -C eth4 adaptive-rx off adaptive-tx off > >> stats-block-usecs: 0 >> sample-interval: 0 >> pkt-rate-low: 0 >> pkt-rate-high: 0 >> >> rx-usecs: 32 > > When you run off "adaptive-rx" you will get 31250 interrupts/sec > (calc 1/(32/10^6) = 31250). > >> rx-frames: 64 > > You likely want to reduce this. > >> rx-usecs-irq: 0 >> rx-frames-irq: 0 >> >> tx-usecs: 8 >> tx-frames: 128 > > These 128 frames also seems too large. > >> tx-usecs-irq: 0 >> tx-frames-irq: 0 >> > [...] I have now updated the settings to: ethtool -c eth4 Coalesce parameters for eth4: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 0 rx-frames: 32 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 16 tx-frames: 32 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 but it makes absolutely no difference, also discard keep coming as before. >>> >>> You can test theory(2) in two ways, via disabling TSO and via >>> adjusting >>> ethtool --show-coalesce|--coalesce (-c|-C). The --coalesce change can >>> change part of theory(1), the NIC wake-up/IRQ interval. here's the current nic settings: ethtool -k eth4 Features for eth4: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: off udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: on tx-gre-csum-segmentation: on tx-ipxip4-segmentation: on tx-ipxip6-segmentation: on tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] tx-udp-segmentation: on fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off rx-all: off tx-vlan-stag-hw-insert: on rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: on [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: on tls-hw-tx-offload: off [fixed] tls-hw-rx-offload: off [fixed] rx-gro-hw: off [fixed] tls-hw-record: off [fixed] > > What is the NIC ring size settings? > > ethtool -g eth4 > ethtool -g eth4 Ring parameters for eth4: Pre-set maximums: RX: 8192 RX Mini: 0 RX Jumbo: 0 TX: 8192 Current hardware settings: RX: 1024 RX Mini: 0 RX Jumbo: 0 TX: 1024 > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer _______________________________________________ Bloat mailing list [email protected] https://lists.bufferbloat.net/listinfo/bloat
