Dear Paul, On Fri, Feb 16, 2024 at 10:18:03PM +0100, Paul Menzel wrote: > > Am 16.02.24 um 15:18 schrieb Martin Kulas: > > > My time-sensitive user space application reads out > > the hardware TX timestamp via poll() and recvmsg(MSG_ERRQUEUE). > > On an Intel E810-XXVDA2 (for SFP), it takes roughly 700 microseconds > > until the TX timestamp is available in user space (Linux 6.7.4). > > This latency of 700 microseconds is far too high for my application. > > For comparison, on an Intel X550 card (Twisted pair), > > my application retrieves the TX timestamp in roughly 30 microseconds > > which is acceptable. > >
[...] > > > > When searching for a solution about this high latency TX timestamp read, > > I found the following thread from 2022: > > > > https://patchwork.ozlabs.org/project/intel-wired-lan/patch/[email protected]/#2963591 > > > > Is this patch already in the current Linux Kernel 6.7.4 or is it pending? > > In the metadata on the patchwork page it says *Accepted*, and it seems to be > commit 1229b33973c7 (ice: Add low latency Tx timestamp read) added to Linux > 6.1-rc1, so it should be in Linux 6.1. > > Please build the Linux kernel from this commit to rule out any regression. > (If that commit actually should support such low times that you require. The > commit message says something about 20 ms before.) After having applied that commit, the TX timestamp read latency stays unacceptable high (in the range of about 700 microseconds), i.e. no regression. Here you find details about the E810-XXVDA2 network card with high Tx timestamp read latency: $ ethtool enp65s0f1 Settings for enp65s0f1: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full 10000baseT/Full 25000baseCR/Full 25000baseSR/Full 1000baseX/Full 10000baseSR/Full 10000baseLR/Full Supported pause frame use: Symmetric Supports auto-negotiation: No Supported FEC modes: None Advertised link modes: 10000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: None Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes $ ethtool -i enp65s0f1 driver: ice version: 6.0.0-rc4-kulas-1229+ firmware-version: 4.40 0x8001c96c 1.3534.0 expansion-rom-version: bus-info: 0000:41:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes $ With an other NIC that supports hardware TX timestamping (Intel X550), the TX timestamp read latency is much smaller than on Intel E810-XXVDA2, i.e. 7 microseconds. I am puzzled that an older Intel network card provides the hardware TX timestamp about 100 times faster than an new Intel network card. Configuration of the X550 network card with low Tx timestamp read latency: $ ethtool eno3 Settings for eno3: Supported ports: [ TP ] Supported link modes: 100baseT/Full 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 100baseT/Full 1000baseT/Full 10000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 0 Transceiver: internal MDI-X: Unknown Supports Wake-on: umbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes $ ethtool -i eno3 driver: ixgbe version: 5.17.12-100.fc34.x86_64 firmware-version: 0x80001685, 22.0.9 expansion-rom-version: bus-info: 0000:02:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes $ Do you find anything suspicious in the configuration? Is it a problem of connector types, i.e. twisted pair vs. direct attach? If you want to experiment by your own, you find a minimum program to read out hardware TX timestamp in my ownCloud space: https://owncloud.gwdg.de/index.php/s/q7F6IuoynzGGZis Which tuning knobs exist that allows me to reduce the latency for reading out the TX timestamp from the network card? Kind regards, Martin
