Re: [Openvpn-devel] [PATCH applied] Re: dco: implement dco support for p2p/client code path
Hi, On Thu, Aug 04, 2022 at 03:30:25PM +0200, Gert Doering wrote: > Test sets failed: 2b 2c 2f. So, this is an interesting one. To trigger this, you need to connect over UDP + IPv6 transport, and then you need to inject a packet into "openvpn with DCO" that is fragmented into two parts, which *both* create an UDP packet after encapsulation which is bigger than 1500 bytes. $ ping -c1 -s 2900 10.194.2.1 does this. Tcpdump on the "tun0" interface shows 15:49:16.433666 IP (tos 0x0, ttl 64, id 18639, offset 0, flags [+], proto ICMP (1), length 1500) ubuntu2004 > 10.194.2.1: ICMP echo request, id 170, seq 1, length 1480 15:49:16.433703 IP (tos 0x0, ttl 64, id 18639, offset 1480, flags [none], proto ICMP (1), length 1448) ubuntu2004 > 10.194.2.1: icmp so this is one "full sized 1500 byte frame", containing the first fragment, and then a 1448 byte frame containing the rest. Connecting with --disable-dco yields this on the outside: 17:09:14.763519 IP6 (flowlabel 0x0082a, hlim 62, next-header Fragment (44) payload length: 1456) ubuntu2004 > phillip: frag (0x277a4245:0|1448) 34646 > 51194: UDP, bad length 1524 > 1440 17:09:14.763548 IP6 (flowlabel 0x0082a, hlim 62, next-header Fragment (44) payload length: 92) ubuntu2004 > phillip: frag (0x277a4245:1448|84) 17:09:14.763576 IP6 (flowlabel 0x0082a, hlim 62, next-header Fragment (44) payload length: 1456) ubuntu2004 > phillip: frag (0x1dec474f:0|1448) 34646 > 51194: UDP, bad length 1472 > 1440 17:09:14.763593 IP6 (flowlabel 0x0082a, hlim 62, next-header Fragment (44) payload length: 40) ubuntu2004 > phillip: frag (0x1dec474f:1448|32) this is "4 packets are sent" (and, subsequently, 4 packets come back). Both "(close to) full inside packets" are too big for an external 1500 byte packet, and get fragmented. Is this ugly? Yes. Is it according to IP specs? Yes. Do I expect this to work? Yes! (t_client excercises this case with the 3000-byte-pings). Now, *with* DCO, we get this: 17:10:43.771215 IP6 (hlim 62, next-header Fragment (44) payload length: 1456) ubuntu2004 > phillip: frag (0xe79d84cc:0|1448) 58755 > 51194: UDP, bad length 1524 > 1440 17:10:43.771246 IP6 (hlim 62, next-header Fragment (44) payload length: 92) ubuntu2004 > phillip: frag (0xe79d84cc:1448|84) (only *two* packets, then only keepalives) And at the same time, the client (!) complains 2022-08-04 17:10:38 net_iface_new: add tun0 type ovpn-dco 2022-08-04 17:10:38 DCO device tun0 opened (so, we're using DCO) 2022-08-04 17:10:43 read UDPv6 [CMSG=50|EMSGSIZE Path-MTU=1500]: Message too long (fd=3,code=90) Now... reducing the ping size to 2500... gert@ubuntu2004:~$ ping -c1 -s 2500 10.194.2.1 2508 bytes from 10.194.2.1: icmp_seq=1 ttl=64 time=126 ms ... actually works. What is sent out by DCO in this case is 3 packets, 2 fragments, and one regular UDP packet 17:12:21.512762 IP6 (hlim 62, next-header Fragment (44) payload length: 1456) ubuntu2004 > phillip: frag (0x12728a31:0|1448) 58755 > 51194: UDP, bad length 1524 > 1440 17:12:21.512806 IP6 (hlim 62, next-header Fragment (44) payload length: 92) ubuntu2004 > phillip: frag (0x12728a31:1448|84) 17:12:21.512864 IP6 (hlim 63, next-header UDP (17) payload length: 1080) ubuntu2004.58755 > phillip.51194: [udp sum ok] UDP, length 1072 ... because the second half of the fragmented ICMP packet now fits into one UDP/1500 byte even after openvpn encapsulation. So my current guess is that the linux kernel sets *some* flag on "non-primary ICMP fragments" (inside) that get carried over to the final encapsulated packet, preventing further fragmentation (outside), but eliciting an "PMTU fail" error instead - which the newly renovated extended socket error handler picks up. In userland :-) To add a data point: gert@ubuntu2004:~$ ping -c1 -s 3500 10.194.2.1 creates *3* packets in the DCO case 17:14:08.780733 IP6 (hlim 62, next-header Fragment (44) payload length: 1456) ubuntu2004 > phillip: frag (0x16acdc72:0|1448) 58755 > 51194: UDP, bad length 1524 > 1440 17:14:08.780757 IP6 (hlim 62, next-header Fragment (44) payload length: 92) ubuntu2004 > phillip: frag (0x16acdc72:1448|84) 17:14:08.780842 IP6 (hlim 63, next-header UDP (17) payload length: 600) ubuntu2004.58755 > phillip.51194: [udp sum ok] UDP, length 592 ... and one error message. In the non-DCO case, this creates *5* packets - 2 1500-inside packet that get outside fragmented to 2+2 and one smaller "leftover" packet. So the 2nd fragment is, again, refusing to be fragmented again... (ping -s 5000 yields 7 packet without DCO, and only 3 with DCO, so a pattern starts to form ;-) ) gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany
Re: [Openvpn-devel] [PATCH applied] Re: dco: implement dco support for p2p/client code path
On 04/08/2022 15:30, Gert Doering wrote: Acked-by: Gert Doering []> Your patch has been applied to the master branch. Wohooo! Great stuff! Thanks to everybody who contributed to this first milestone! However, the party is not over yet :-) Cheers, commit b6f7b285767e66f5cbd3854cf0ff918e87b31202 Author: Antonio Quartulli Date: Thu Aug 4 09:14:01 2022 +0200 dco: implement dco support for p2p/client code path Signed-off-by: Antonio Quartulli Acked-by: Gert Doering Message-Id: <20220804071401.12410-...@unstable.cc> URL: https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg24798.html Signed-off-by: Gert Doering -- kind regards, Gert Doering -- Antonio Quartulli ___ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel
[Openvpn-devel] [PATCH applied] Re: dco: implement dco support for p2p/client code path
Acked-by: Gert Doering
After all the preliminary infrastructure building, *this* is the beginning
of the real thing :-)
I have tested
- full set of server side tests, without --enable-dco
(this system does not have kernel DCO, so it does not matter)
--> all works
- full set of client side tests, with --enable-dco, but no kernel DCO
("existing setup")
--> all works
- full set of client side tests, with --enable-dco AND kernel DCO
(wohoo!)
--> some test instances disable DCO (like, SOCKS or HTTP proxy, or
TAP mode), and the fallback works ("pings succeed")
1b:openvpn.log Note: --http-proxy disables data channel offload.
1c:openvpn.log Note: --http-proxy disables data channel offload.
1d:openvpn.log Note: --socks-proxy disables data channel offload.
1e:openvpn.log Note: --socks-proxy disables data channel offload.
1z:openvpn.log Note: Using compression disables data channel offload.
2a:openvpn.log Note: cipher 'BF-CBC' in --data-ciphers is not
supported by ovpn-dco, disabling data channel offload.
2d:openvpn.log Note: --socks-proxy disables data channel offload.
2e:openvpn.log Note: --socks-proxy disables data channel offload.
2z:openvpn.log Note: Using compression disables data channel offload.
3z:openvpn.log Note: Using compression disables data channel offload.
4:openvpn.log Note: dev-type not tun, disabling data channel offload.
4a:openvpn.log Note: dev-type not tun, disabling data channel offload.
4b:openvpn.log Note: dev-type not tun, disabling data channel offload.
6:openvpn.log Note: --fragment disables data channel offloa .
8:openvpn.log Note: Using compression disables data channel offload.
9:openvpn.log Note: dev-type not tun, disabling data channel offload.
23:openvpn.log Note: --data-cipher-fallback with cipher 'BF-CBC'
disables data channel offload.
23a:openvpn.log Note: Using compression disables data channel offload.
23s:openvpn.log Note: --data-cipher-fallback with cipher 'BF-CBC'
disables data channel offload.
24:openvpn.log Note: Using compression disables data channel offload.
24a:openvpn.log Note: Using compression disables data channel offload.
--> other instances claim to are using DCO ("ip -d link show"
shows "ovpn-dco") *and* packets are moved, so I guess it's
using DCO...
these tests include "normal --client clients", "p2p --secret",
"p2p --tls-secret", and "p2p --tls-secret with P2P NCP", using
varying ciphers (-> BF-CBC/none forcing non-DCO, etc.)
Test sets succeeded: 1 1a 1b 1c 1d 1e 1z 2 2a 2d 2e 2z 3 3z 4 4a 4b 5 6
8 9 23 23a 23s 24 24a.
Test sets failed: 2b 2c 2f.
The 3 failures (2b, 2c, 2f) are all "IPv6 UDP fragments" (ping -s 3000,
encapsulated in IPv6 UDP), which needs closer investigation. This works
on a "--disable-dco" build, but the whole topic of UDP fragmentation
is "outside OpenVPN", so this is not something a patch to OpenVPN
can affect. tcpdump on an intermediate host can see outgoing fragments
in the DCO case, but no replies - different from the non-DCO case, so
this is going to be an interesting root cause hunt...
- I have not done performance tests, because the current test
environment is not really suited for it yet (server instances
are all non-DCO)
Plus, stared at the code and discussed with Antonio on IRC :-) -
(especially the process_outgoing_link() change confused me a bit - the
obvious answer to this is "these are control channel packets, which are
still created by userland, but the design requires to avoid accessing
the socket directly, so send to DCO module, and that one forwards").
I removed one spurious blank line from dco_p2p_add_new_peer().
Your patch has been applied to the master branch.
commit b6f7b285767e66f5cbd3854cf0ff918e87b31202
Author: Antonio Quartulli
Date: Thu Aug 4 09:14:01 2022 +0200
dco: implement dco support for p2p/client code path
Signed-off-by: Antonio Quartulli
Acked-by: Gert Doering
Message-Id: <20220804071401.12410-...@unstable.cc>
URL:
https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg24798.html
Signed-off-by: Gert Doering
--
kind regards,
Gert Doering
___
Openvpn-devel mailing list
Openvpn-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openvpn-devel
