I initially mistook this for a traffic control issue, but after stripping the test beds down to just the MACSec component, I can still replicate the issue. After approximately 5TB of transfer / 4 billion packets over a MACSec link it stops passing traffic. I have replicated this now on both Ubuntu Server 18.04 with their patched 4.15 kernel and Gentoo with a vanilla 4.18.13 kernel.
As noted before, my test setup consists of two machines with a direct ethernet connection to each other. MACSec is setup on the link, along with a /30 network. I then hammer the link with iperf3 in both directions. On a gigabit link it takes me one to two days to trip the bug. Nothing is logged to dmesg. If I remove and re-add the MACSec link, or reboot the machines traffic flow resumes. I can replicate on physical hardware or in VMs. How should I proceed to help diag and correct this? To replicate, setup two machines with a direct ethernet connection. If simulating in ESXi setup a dedicated vSwitch, allow promiscuous, forged and MAC changes should be enabled, MTU increased to 9000, then setup a dedicated port group and VLAN to simulate the direct connection. I use the following script to setup the MACSec connection, adjusting keys, rxmac, interface and IPs as appropriate: ----------- The script I used on each host (keys, rxmacs and IPs updated as appropriate): #!/bin/bash # Interfaces: # dif = Egress physical interface (Dest) # eif = Encrypted interface dif=ens224 eif=macsec0 # MACSec Keys: # txkey = Transmit (Local) key # rxkey = Receive (Remote) key # rxmac = Receive (Remote) MAC addy txkey=60995924232808431491190820961556 rxkey=87345530111733181210202106249824 rxmac=00:0c:29:c5:95:df # Clear any existing IP config ifconfig $dif 0.0.0.0 # Bring up macsec: echo "* Enable MACSec" modprobe macsec ip link add link "$dif" "$eif" type macsec ip macsec add "$eif" tx sa 0 pn 1 on key 02 "$txkey" ip macsec add "$eif" rx address "$rxmac" port 1 ip macsec add "$eif" rx address "$rxmac" port 1 sa 0 pn 1 on key 01 "$rxkey" ip link set "$eif" type macsec encrypt on # Bring up the interfaces: echo "* Light tunnel NICS" ip link set "$dif" up ip link set "$eif" up # Set IP ifconfig $eif 192.168.211.1/30 Once you can ping across the link, use iperf3 or a similar network stress tool to flood the link with traffic in both directions and wait for the bug to trigger. Josh Coombs
