Hi,
IMHO, Interrupt coalescence does not work as advertised with
the igb and e1000e drivers. Interrupt coalescence seems to not use
properly the 'rx-usecs' parameter. Moreover, when running at 100 Mb/s,
it seems to be scaled by a factor 10, resulting in huge packet
latency.
The Broadcom tg3 driver has a default time window for
interrupt coalescence of about 20 us, as per ethtool. With UDP
traffic, a 1 Gb/s, I measure a time window of around 40 us. At 100
Mb/s, I measure a time window of around 60 us in the worse case, in
other cases it's below 20 us.
The X710/i40e driver has a default time window of 25 us. With
UDP traffic, I measure a time window of around 20 us, at 10 Gb/s
(sorry, can't change speed).
Both the igb driver, the default time window for interrupt
coalescence of about 3 us, as per ethtool (see below).
With UDP traffic, at 1 Gb/s, I measure a time window of around
40 us.
At 100 Mb/s, I measure a time window of around 200-250 us (see
below).
The e1000e driver has the same behaviour as igb.
If in igb, I use ethtool to change the time window for
interrupt coalescence to 1 us (i.e., the smallest value), it does not
seem to help. With rx-usecs set to 1, at 100 Mb/s, I still measure
around 200-250 us, and at 1 Gb/s, 40 us. I.e., reducing the parameter
does not change anything.
As igb and e1000e don't seem to be using the rx-usecs
parameter properly, there seems to be no way to set interrupt
coalescence to a sensible value at 100 Mb/s (let say, 20-40 us).
So, IMHO, buggy.
Setup :
-----
Linux kernel : 4.9.82
igb driver : 5.3.5.18 & 5.4.0-k
firmware-version: 3.16, 0x800004ff, 1.304.0
e1000e driver : 3.2.6-k
firmware-version: 5.6-10
Speed is changed with :
# ethtool -s eth6 speed 100 duplex full
Speed change was validated with ethtool and running netperf
Interrupt Coal :
--------------
Default settings. With the default setting, the max int-coal
time window is supposed to be 3 usec.
# ethtool -c eth6
Coalesce parameters for eth6:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 128
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
rx-usecs was changed with :
# ethtool -C eth6 rx-usecs 1
Received packet log :
-------------------
I run a stream of UDP packet, each packet is 1000 byte, log of
receive times is below. In the received packet time log, I can see
groups of packets batched closely together (separated by 2 usec,
indicating a speed of 4 Gb/s, i.e. clearly coalesced), and those
groups are separated by 200-250 us.
Please find below the packet log. First column is received
time from the kernel in nanoseconds. Second column is time difference
to previous received packet, i.e. the gap between two received
packets in nanoseconds.
1524864423.932657212 ; 226658
1524864423.932659491 ; 2279
1524864423.932661023 ; 1532
1524864423.932662308 ; 1285
1524864423.932923593 ; 261285
1524864423.932927914 ; 4321
1524864423.933147424 ; 219510
1524864423.933149500 ; 2076
1524864423.933407932 ; 258432
1524864423.933412290 ; 4358
1524864423.933413758 ; 1468
1524864423.933415107 ; 1349
1524864423.933637323 ; 222216
1524864423.933639505 ; 2182
1524864423.933640854 ; 1349
1524864423.933899512 ; 258658
1524864423.934130315 ; 230803
1524864423.934132701 ; 2386
1524864423.934134096 ; 1395
1524864423.934135391 ; 1295
1524864423.934377340 ; 241949
1524864423.934381582 ; 4242
1524864423.934382997 ; 1415
1524864423.934628593 ; 245596
1524864423.934885071 ; 256478
1524864423.934886928 ; 1857
1524864423.934888129 ; 1201
1524864423.934889348 ; 1219
1524864423.934890530 ; 1182
1524864423.935107683 ; 217153
1524864423.935110140 ; 2457
1524864423.935111242 ; 1102
1524864423.935367980 ; 256738
1524864423.935371950 ; 3970
1524864423.935373136 ; 1186
1524864423.935600854 ; 227718
1524864423.935855419 ; 254565
1524864423.935857871 ; 2452
1524864423.935859196 ; 1325
1524864423.935860454 ; 1258
1524864423.935861796 ; 1342
1524864423.936087278 ; 225482
1524864423.936089122 ; 1844
1524864423.936366475 ; 277353
1524864423.936577632 ; 211157
1524864423.936580055 ; 2423
1524864423.936581466 ; 1411
1524864423.936582828 ; 1362
1524864423.936584173 ; 1345
1524864423.936848161 ; 263988
1524864423.936852586 ; 4425
1524864423.936853977 ; 1391
1524864423.937069593 ; 215616
1524864423.937071769 ; 2176
1524864423.937073050 ; 1281
1524864423.937333677 ; 260627
1524864423.937557332 ; 223655
1524864423.937561290 ; 3958
1524864423.937562748 ; 1458
1524864423.937564110 ; 1362
1524864423.937826533 ; 262423
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
