On 6 Nov 2020, at 12:45, Toke Høiland-Jørgensen wrote:
"Thomas Rosenstein" <[email protected]> writes:
On 6 Nov 2020, at 12:18, Jesper Dangaard Brouer wrote:
On Fri, 06 Nov 2020 10:18:10 +0100
"Thomas Rosenstein" <[email protected]> wrote:
I just tested 5.9.4 seems to also fix it partly, I have long
stretches where it looks good, and then some increases again.
(3.10
Stock has them too, but not so high, rather 1-3 ms)
That you have long stretches where latency looks good is interesting
information. My theory is that your system have a periodic
userspace
process that does a kernel syscall that takes too long, blocking
network card from processing packets. (Note it can also be a kernel
thread).
The weird part is, I first only updated router-02 and pinged to
router-04 (out of traffic flow), there I noticed these long stretches
of
ok ping.
When I updated also router-03 and router-04, the old behaviour kind
of
was back, this confused me.
Could this be related to netlink? I have gobgpd running on these
routers, which injects routes via netlink.
But the churn rate during the tests is very minimal, maybe 30 - 40
routes every second.
Otherwise we got: salt-minion, collectd, node_exporter, sshd
collectd may be polling the interface stats; try turning that off?
I can, but shouldn't that also influence iperf3 performance then?
Another theory is the NIC HW does strange things, but it is not very
likely. E.g. delaying the packets before generating the IRQ
interrupt,
which hide it from my IRQ-to-softirq latency tool.
A question: What traffic control qdisc are you using on your system?
kernel 4+ uses pfifo, but there's no dropped packets
I have also tested with fq_codel, same behaviour and also no
weirdness
in the packets queue itself
kernel 3.10 uses mq, and for the vlan interfaces noqueue
Do you mean that you only have a single pfifo qdisc on kernel 4+? Why
is
it not using mq?
oh, actually, I just noticed that's a remnant of the previous tests, I
had
net.core.default_qdisc = fq_codel
in the sysctl.conf... so disregard my previous wrong info
so all kernel by default look like that, mq + pfifo_fast:
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth1 root
qdisc pfifo_fast 0: dev eth1 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth2 root
qdisc pfifo_fast 0: dev eth2 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth3 root
qdisc pfifo_fast 0: dev eth3 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth4 root
qdisc pfifo_fast 0: dev eth4 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :9 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :a bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :b bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :c bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :d bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :e bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :f bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :10 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :11 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :12 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :13 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :14 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :15 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :16 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :17 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :18 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc noqueue 0: dev eth4.2300 root refcnt 2
qdisc noqueue 0: dev eth4.2503 root refcnt 2
Was there anything in the ethtool stats?
no, just rx / tx increase, not a single err
-Toke
_______________________________________________
Bloat mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/bloat
