https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295263
Bug ID: 295263
Summary: jails with VNET and LRO enabled interface causing
severe performance issues
Product: Base System
Version: 15.0-RELEASE
Hardware: Any
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
After having upgraded our servers from FreeBSD 13.5 to 15.0 we are seeing
severe performance issues when sending data to processes running inside the
jails.
Disabling LRO ("ifconfig lagg0 -lro") make things to fast again. With LRO
enabled it's so slow to basically be useless...
Test setup: Servers with 2x10Gbps Intel X710 "ixl" interfaces configured as a
"failover" lagg. A jail with VNET enabled. Runnings "iperf3 -s" inside the jail
and then sending data to it "iperf3 -c <ip>" from two machines:
A) Workstation connected with 1Gbps ethernet från another VLAN
B) Server (Dell R740) Sending data locally on the same server
C) Server (Dell R710) Sending data between two similar servers on same VLAN
from within a jail on each server
Without LRO enabled ("ifconfig lagg0 -lro"):
A: 936 Mbit/s
B: 6.0 Gbit/s
C: 4.8 Gbit/s
With LRO enabled ("ifconfig lagg0 lro"):
A: 21.7 Mbit/s
B: 0.00 bits/s (a few packets seems to get thru)
C: 0.00 bits/s (-"-)
A (Remotely):
# iperf3 -c 130.236.8.70
Connecting to host 130.236.8.70, port 5201
[ 5] local 130.236.16.244 port 33998 connected to 130.236.8.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 950 Mbits/sec 0 491 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 491 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 935 Mbits/sec 0 491 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec 0 491 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 930 Mbits/sec 0 491 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 934 Mbits/sec 0 491 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 0 491 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 491 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 934 Mbits/sec 0 491 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 936 Mbits/sec 0 491 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec receiver
iperf Done.
# iperf3 -c 130.236.8.70
Connecting to host 130.236.8.70, port 5201
[ 5] local 130.236.16.244 port 54966 connected to 130.236.8.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.00 MBytes 16.8 Mbits/sec 202 2.83 KBytes
[ 5] 1.00-2.00 sec 2.25 MBytes 18.9 Mbits/sec 246 2.83 KBytes
[ 5] 2.00-3.00 sec 2.12 MBytes 17.8 Mbits/sec 252 2.83 KBytes
[ 5] 3.00-4.00 sec 2.12 MBytes 17.8 Mbits/sec 262 2.83 KBytes
[ 5] 4.00-5.00 sec 2.12 MBytes 17.8 Mbits/sec 250 2.83 KBytes
[ 5] 5.00-6.00 sec 2.25 MBytes 18.9 Mbits/sec 240 2.83 KBytes
[ 5] 6.00-7.00 sec 4.25 MBytes 35.7 Mbits/sec 196 2.83 KBytes
[ 5] 7.00-8.00 sec 3.38 MBytes 28.3 Mbits/sec 240 2.83 KBytes
[ 5] 8.00-9.00 sec 2.75 MBytes 23.1 Mbits/sec 270 2.83 KBytes
[ 5] 9.00-10.00 sec 2.62 MBytes 22.0 Mbits/sec 222 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 25.9 MBytes 21.7 Mbits/sec 2380 sender
[ 5] 0.00-10.00 sec 25.8 MBytes 21.6 Mbits/sec receiver
iperf Done.
B (Locally):
# ifconfig lagg0 lro
# iperf3 -c 130.236.8.70
Connecting to host 130.236.8.70, port 5201
[ 5] local 130.236.8.47 port 30070 connected to 130.236.8.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 0.00 Bytes 0.00 bits/sec 14 5.66 KBytes
[ 5] 1.06-2.02 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
[ 5] 2.02-3.00 sec 0.00 Bytes 0.00 bits/sec 4 2.83 KBytes
[ 5] 3.00-4.01 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
[ 5] 4.01-5.00 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
[ 5] 6.00-7.06 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
[ 5] 7.06-8.05 sec 0.00 Bytes 0.00 bits/sec 7 1.41 KBytes
[ 5] 8.05-9.06 sec 0.00 Bytes 0.00 bits/sec 7 2.83 KBytes
[ 5] 9.06-10.06 sec 0.00 Bytes 0.00 bits/sec 6 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 0.00 Bytes 0.00 bits/sec 68 sender
[ 5] 0.00-10.06 sec 0.00 Bytes 0.00 bits/sec receiver
iperf Done.
# ifconfig lagg0 -lro
# iperf3 -c 130.236.8.70
Connecting to host 130.236.8.70, port 5201
[ 5] local 130.236.8.47 port 30988 connected to 130.236.8.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 688 MBytes 5.42 Gbits/sec 181 1.75 MBytes
[ 5] 1.06-2.04 sec 578 MBytes 4.99 Gbits/sec 0 1.85 MBytes
[ 5] 2.04-3.02 sec 610 MBytes 5.18 Gbits/sec 0 1.92 MBytes
[ 5] 3.02-4.06 sec 556 MBytes 4.52 Gbits/sec 0 1.98 MBytes
[ 5] 4.06-5.02 sec 545 MBytes 4.75 Gbits/sec 0 2.02 MBytes
[ 5] 5.02-6.00 sec 470 MBytes 4.00 Gbits/sec 0 2.04 MBytes
[ 5] 6.00-7.00 sec 660 MBytes 5.55 Gbits/sec 0 2.06 MBytes
[ 5] 7.00-8.00 sec 621 MBytes 5.21 Gbits/sec 0 2.06 MBytes
[ 5] 8.00-9.00 sec 538 MBytes 4.51 Gbits/sec 0 2.06 MBytes
[ 5] 9.00-10.01 sec 519 MBytes 4.31 Gbits/sec 0 2.06 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 5.65 GBytes 4.85 Gbits/sec 181 sender
[ 5] 0.00-10.01 sec 5.65 GBytes 4.85 Gbits/sec receiver
iperf Done.
Network configuration:
/etc/rc.conf:
ifconfig_ixl0="up"
ifconfig_ixl2="up"
cloned_interfaces="lagg0 vlan1601"
ifconfig_lagg0="laggproto failover laggport ixl0 laggport ixl1 laggport ixl2
laggport ixl3 130.236.8.47 netmask 255.255.255.224 "
ifconfig_lagg0_ipv6="inet6 2001:6b0:17:2400::8:47/64 "
defaultrouter="130.236.8.33"
ipv6_defaultrouter="2001:6b0:17:2400::1"
pf_enable="YES"
pflog_enable="YES"
ifconfig_vlan1601="vlandev lagg0 vlan 1601 up"
ifconfig_ixl1="up"
ifconfig_ixl3="up"
/etc/sysctl.conf:
net.inet.ip.forwarding = 0
net.link.bridge.pfil_onlyip = 0
net.link.bridge.pfil_bridge = 0
net.link.bridge.pfil_member = 0
--
You are receiving this mail because:
You are the assignee for the bug.
