Reporting a failure mode in openvpn3-linux where a split-tunnel profile
with a large pushed route set cannot connect. openvpn3-service-netcfg's
per-operation netlink socket churn exceeds the client's fixed
Establish() timeout. Reproduced identically on v26-3 and v27.1.
ENVIRONMENT
openvpn3-linux v26-3 and v27.1 (both affected)
OpenVPN core 3.11.5 / 3.11.7
OS Ubuntu 24.04, systemd + D-Bus
Kernel 6.18.33.2 (WSL2)
Profile split-tunnel, ~900 pushed IPv4 prefixes
SYMPTOM
Authentication succeeds, the tunnel negotiates, then:
Virtual device '<id>' registered on /net/openvpn/v3/netcfg/<id>
Adding IP Address 172.x.y.z/24 gw 172.x.y.1 ipv6: no
Setting remote IP address to <server> ipv6: no
Error calling NetCfgDevice::Establish(): [Proxy::Client(
'net.openvpn.v3.netcfg', ..., 'Establish')] Timeout was reached
Failed configuring TUN device (TUN_IFACE_CREATE)
Five seconds later netcfg finishes the work the caller already gave up
on and rolls it back, logging ~900 of:
Error while executing NetlinkRoute4(add: 0) tun0: -2
Error while executing NetlinkAddr4(add: 0) tun0: -2
Error while executing NetlinkLinkSet tun0 mtu 1500: -1
Those are teardown, not the cause. The tun itself is created fine --
ioctl(TUNSETIFF) returns 0 in 221us.
ROOT CAUSE
netcfg allocates, binds, uses and closes a fresh AF_NETLINK socket for
every single operation. From strace, this unit repeats once per route:
socket(AF_UNIX, SOCK_DGRAM) -> ioctl(SIOCGIFINDEX) -> close
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE)
-> setsockopt(SO_SNDBUF) -> setsockopt(SO_RCVBUF)
-> bind() <-- dominant cost
-> getsockname -> sendmsg -> recvmsg -> close
On a typical kernel bind() is microseconds here and nobody notices. In
this environment bind() costs ~10ms, so the fixed per-operation
overhead becomes the whole story:
901 bind() calls mean 9.1ms max 10.8ms total 8.183s
Within the 5s Establish() window:
507 bind() calls
4.601s in bind() alone
4.939s total worker syscall time
-> 93% of the budget is bind()
~900 routes x ~10ms is ~9s against a 5s deadline. netcfg completed 507
of them before the caller aborted.
Note this is not netlink-specific slowness in the environment. AF_UNIX
bind() measures the same ~10ms and AF_INET bind() ~21ms, while socket()
creation is free. The point is that netcfg pays a fixed per-socket
setup cost ~900 times where once would do.
For contrast, on the same host, installing the same route set through a
single reused socket (ip -batch) takes ~0.4s, roughly 20x faster.
Individually via separate "ip route add" invocations it is ~0.9s per 45
routes.
SUGGESTED FIXES
1. Reuse one netlink socket for the duration of an Establish(), or for
the lifetime of the NetCfgDevice, instead of one per operation. This
removes the failure entirely and is a straight win everywhere --
~900 socket create/bind/close cycles per connection is pure overhead
even when bind() is cheap.
2. The Establish() D-Bus call timeout appears to be a fixed 5s with no
override, so a slower-than-anticipated environment has no escape
hatch. Consider scaling it with the number of queued networks, or
making it configurable.
SEPARATE BUG: SEGFAULT ON TIMED-OUT CALLER (v27.1)
When the caller gives up, v27.1 netcfg dereferences the D-Bus unique
name of the process that has already exited, and crashes:
** ERROR ** Async call failed: [DBus::Credentials::GetPID] Failed
to retrieve process ID for bus name ':1.75': Could not get PID of
name ':1.75': no such name
error domain: net.openvpn.gdbuspp
kernel: pool[20958]: segfault at 5e2123c54d3d ip 00005e24841cca58
sp 0000775b767fb250 error 4 in openvpn3-service-netcfg[...]
kernel: openvpn3-service-netcfg: pool: potentially unexpected
fatal signal 11
Looks like a use-after-free or missing existence check on the caller's
bus name in the async error path. v26-3 handles the same timeout
gracefully and only logs the netlink errors, which is why the crash is
v27.1-specific while the underlying timeout affects both.
libgdbuspp3 3-3.
WORKAROUND FOR ANYONE HITTING THIS
Append "route-nopull" to the profile so Establish() only creates the
tun and sets the address, MTU and link state -- about 5 netlink ops,
well inside 5s. Then install the pushed routes separately through one
batched netlink socket with ip -batch. Capture the route list from a
normal connect attempt with:
journalctl --since -5min | grep -oE 'Adding network [0-9./]+' \
| awk '{print $3}' | sort -u
Caution: check whether the VPN server's own address falls inside any
pushed prefix. If it does, a protective /32 host route via the physical
gateway must exist before the tunnel routes go in, or the transport
deadlocks.
_______________________________________________
Openvpn-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/openvpn-devel
