Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces

[Christoph Loesch]

Now I've read enough (rather old and also pretty recent) posts about that topic 
and I want to help finally fix this issue.

Not just because it bugs our network too after we optimized/simplified our 
router-setup but also because it seems over the years many users seem to 
beaffected. Until now every thread I've read ends with an open question rather 
than a solution.

In Jason's post here from 2017 I found his script that should reproduce 
theissue, I tried it and could not reproduce the issue with that script..
https://lists.zx2c4.com/pipermail/wireguard/2017-November/002092.html
To be honest I do not fully understand it to modify it to reproduce the issue 
in my environment.

Here I posted one example and the (temporary) solution by deleting the route 
that wireguard had set and re-creating the same route but defining the 
(correct) IP as src.
https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html

How could I help to fix this issue?
If it helps, I could give full access to a test-router which has the same setup 
as all other routers in our network.

Kind regards


Am 23.11.2021 um 14:52 schrieb Christoph Loesch:

Hi,

Is this related to my issue with using wrong source ip on outgoing connections?
(https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)

Which bug would that be? Would be interesting in more detail to understand the 
issue.
I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, 
since this issue is on only 5 devices from over 20 devices where all are configured same.

The configuration or better all routes do not even have any metric defined so I 
am unsure what to add/remove to change the behaviour.
Also in my configuration there are multiple interfaces 
(br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default 
routes.
Comparing perfectly fine working devices to one of those five devices which 
have this issue I am not able to find a difference in IP-config or routes.
Even kernel and firmware versions are all same.

The only difference I see is while looking at tcpdump where a simple pingto the 
wireguard server's internal IP (172.27.0.1/24 in my case) originates with the 
public IP of the device.

If it is a known bug, it would be nice to know what (and if) I could try to fix 
and possibly confirm the bug. Or even better beeing able to reproduce the issue.

Thanks and kind regards,
Christoph

Am 19.11.2021 um 14:10 schrieb 曹煜:

Hi,
As Jason said in mailing list
(https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
years ago, if wireguard reply the client with wrong src ip, then it is
a bug. And now I'm trying to set up a minimal configuration.

Server info:

Ubuntu 20.04 in VMware with two Host-only network adapters:

wacke@Ubuntu:/etc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

wacke@Ubuntu:/etc$ uname -a
Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

wacke@Ubuntu:/etc$ sudo ip route show
169.254.0.0/16 dev ens33 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
metric 100
192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
metric 101
192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1

wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
ListenPort = 5999

[Peer]
PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
AllowedIPs = 192.168.199.2/32
PersistentKeepalive = 25

[Peer]
PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
AllowedIPs = 192.168.199.3/32
PersistentKeepalive = 25

wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
wacke@Ubuntu:/etc$ sudo wg show
interface: wg0
    public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
    private key: (hidden)
    listening port: 5999

peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
    endpoint: 192.168.187.1:35560
    allowed ips: 192.168.199.2/32
    transfer: 2.89 KiB received, 1.80 KiB sent
    persistent keepalive: every 25 seconds

peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
    allowed ips: 192.168.199.3/32
    persistent keepalive: every 25 seconds

Client info:

HOME-Server:/NAS/Software/Software # uname -a
Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux

HOME-Server:/NAS/Software/Software # cat wg0.conf
[Interface]
PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=

[Peer]
PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
Endpoint = 192.168.187.128:5999
AllowedIPs = 192.168.199.0/24
PersistentKeepalive = 25

HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
HOME-Server:/NAS/Software/Software # ip link set wg0 up
HOME-Server:/NAS/Software/Software # wg show
interface: wg0
    public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
    private key: (hidden)
    listening port: 53058

peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
    endpoint: 192.168.187.128:5999
    allowed ips: 192.168.199.0/24
    transfer: 0 B received, 59.26 KiB sent
    persistent keepalive: every 25 seconds


tcpdump of client side:

HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
snapshot length 262144 bytes
22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
flags [none], proto UDP (17), length 176)
      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

tcpdump of server side:

wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
capture size 262144 bytes
22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
[none], proto UDP (17), length 120)
      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel

As the tcpdump log showed, the wireguard server always chose the ip
with lowest metric as source ip (192.168.187.129) to reply to the
client (while the client tried to connect 192.168.187.128).

Hope the info above can help to debug. Many thanks.