I let Jean know that we have an internal HPE team and he should be going through them for this issue.
Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Jean Tourrilhes [mailto:j...@labs.hpe.com] Sent: Friday, April 27, 2018 4:32 PM To: e1000-devel@lists.sourceforge.net Subject: [E1000-devel] Interrupt coalescence issues with igb and e1000e drivers Hi, IMHO, Interrupt coalescence does not work as advertised with the igb and e1000e drivers. Interrupt coalescence seems to not use properly the 'rx-usecs' parameter. Moreover, when running at 100 Mb/s, it seems to be scaled by a factor 10, resulting in huge packet latency. The Broadcom tg3 driver has a default time window for interrupt coalescence of about 20 us, as per ethtool. With UDP traffic, a 1 Gb/s, I measure a time window of around 40 us. At 100 Mb/s, I measure a time window of around 60 us in the worse case, in other cases it's below 20 us. The X710/i40e driver has a default time window of 25 us. With UDP traffic, I measure a time window of around 20 us, at 10 Gb/s (sorry, can't change speed). Both the igb driver, the default time window for interrupt coalescence of about 3 us, as per ethtool (see below). With UDP traffic, at 1 Gb/s, I measure a time window of around 40 us. At 100 Mb/s, I measure a time window of around 200-250 us (see below). The e1000e driver has the same behaviour as igb. If in igb, I use ethtool to change the time window for interrupt coalescence to 1 us (i.e., the smallest value), it does not seem to help. With rx-usecs set to 1, at 100 Mb/s, I still measure around 200-250 us, and at 1 Gb/s, 40 us. I.e., reducing the parameter does not change anything. As igb and e1000e don't seem to be using the rx-usecs parameter properly, there seems to be no way to set interrupt coalescence to a sensible value at 100 Mb/s (let say, 20-40 us). So, IMHO, buggy. Setup : ----- Linux kernel : 4.9.82 igb driver : 5.3.5.18 & 5.4.0-k firmware-version: 3.16, 0x800004ff, 1.304.0 e1000e driver : 3.2.6-k firmware-version: 5.6-10 Speed is changed with : # ethtool -s eth6 speed 100 duplex full Speed change was validated with ethtool and running netperf Interrupt Coal : -------------- Default settings. With the default setting, the max int-coal time window is supposed to be 3 usec. # ethtool -c eth6 Coalesce parameters for eth6: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 3 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 0 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 128 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 rx-usecs was changed with : # ethtool -C eth6 rx-usecs 1 Received packet log : ------------------- I run a stream of UDP packet, each packet is 1000 byte, log of receive times is below. In the received packet time log, I can see groups of packets batched closely together (separated by 2 usec, indicating a speed of 4 Gb/s, i.e. clearly coalesced), and those groups are separated by 200-250 us. Please find below the packet log. First column is received time from the kernel in nanoseconds. Second column is time difference to previous received packet, i.e. the gap between two received packets in nanoseconds. 1524864423.932657212 ; 226658 1524864423.932659491 ; 2279 1524864423.932661023 ; 1532 1524864423.932662308 ; 1285 1524864423.932923593 ; 261285 1524864423.932927914 ; 4321 1524864423.933147424 ; 219510 1524864423.933149500 ; 2076 1524864423.933407932 ; 258432 1524864423.933412290 ; 4358 1524864423.933413758 ; 1468 1524864423.933415107 ; 1349 1524864423.933637323 ; 222216 1524864423.933639505 ; 2182 1524864423.933640854 ; 1349 1524864423.933899512 ; 258658 1524864423.934130315 ; 230803 1524864423.934132701 ; 2386 1524864423.934134096 ; 1395 1524864423.934135391 ; 1295 1524864423.934377340 ; 241949 1524864423.934381582 ; 4242 1524864423.934382997 ; 1415 1524864423.934628593 ; 245596 1524864423.934885071 ; 256478 1524864423.934886928 ; 1857 1524864423.934888129 ; 1201 1524864423.934889348 ; 1219 1524864423.934890530 ; 1182 1524864423.935107683 ; 217153 1524864423.935110140 ; 2457 1524864423.935111242 ; 1102 1524864423.935367980 ; 256738 1524864423.935371950 ; 3970 1524864423.935373136 ; 1186 1524864423.935600854 ; 227718 1524864423.935855419 ; 254565 1524864423.935857871 ; 2452 1524864423.935859196 ; 1325 1524864423.935860454 ; 1258 1524864423.935861796 ; 1342 1524864423.936087278 ; 225482 1524864423.936089122 ; 1844 1524864423.936366475 ; 277353 1524864423.936577632 ; 211157 1524864423.936580055 ; 2423 1524864423.936581466 ; 1411 1524864423.936582828 ; 1362 1524864423.936584173 ; 1345 1524864423.936848161 ; 263988 1524864423.936852586 ; 4425 1524864423.936853977 ; 1391 1524864423.937069593 ; 215616 1524864423.937071769 ; 2176 1524864423.937073050 ; 1281 1524864423.937333677 ; 260627 1524864423.937557332 ; 223655 1524864423.937561290 ; 3958 1524864423.937562748 ; 1458 1524864423.937564110 ; 1362 1524864423.937826533 ; 262423 ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
