[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
please reopen if this is still an issue ** Changed in: systemd (Ubuntu) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
It turns out the fix for this issue was backported to systemd v240: https://github.com/systemd/systemd-stable/pull/37 I performed a release upgrade on one of our affected servers, bringing it up from ubuntu 18.04 to ubuntu 19.04 (which uses systemd v240), and I can confirm that the peers are being configured correctly now. So this issue affects ubuntu 18.04 LTS but not any later supported releases. 18.10 was also affected but it's EOL. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
I think the underlying problem is improper fragmentation of netlink messages sent to the WireGuard device by systemd v237 in the set_wireguard_interface function: https://github.com/systemd/systemd/blob/v237/src/network/netdev/wireguard.c#L107 Appending netlink message data can fail if the message size limit has been exceeded. This can happen if there are too many peers or ip masks in the netdev file, and the v237 code doesn't seem to handle this properly. It's supposed to split the data up into message fragments, but instead it can end up writing incoherent data to the netlink socket or end up in an infinite loop. This issue was fixed in systemd v241 by reworking the code over a few commits: https://github.com/systemd/systemd/pull/11418 https://github.com/systemd/systemd/pull/11580 (this fixed issues with the first PR) I found some comments (now resolved) on one of the commits illuminating: https://github.com/systemd/systemd/pull/11418/commits/e1f717d4a02e15ae11a191dd4962b2f4d117678d Mic92 on 2019-01-15: > The idea is that netlink's messages are limited in size. If an interface has many peers, addresses or ip masks then the configuration might not fit into one message and has to be split across different messages. yuwata on 2019-01-15: > Yeah. I guess there was some bug in the cancellation logic, and it causes infinite loop with the magic number 23. The infinite loop with 23 peers yuwata mentions is a reference to Leonid's bug report from January: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811149 I expect that backporting these fixes from v241 to bionic's systemd v237 branch would resolve both my issue and the issue reported by Leonid. I realize this is a non-trivial change and there's a regression risk. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
I now believe the dmesg complaint in my last comment to be a separate issue. A fix for it was backported to systemd v238 in this commit: https://github.com/systemd/systemd- stable/commit/7db3fe08c5eb83584f3a3d356876b4acaa797585#diff- f29d1bfc98e548dc0eb497c3d17cbefa It was not backported to systemd v237: https://github.com/systemd/systemd- stable/commits/v237-stable/src/network/netdev/wireguard.c -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853956] Re: 34 wireguard peers result in invalid peer configuration
On two systems with 33 peers I noticed that this shows up in dmesg after a reboot: netlink: 'systemd-network': attribute type 5 has an invalid length. These lines also show up whenever I run `sudo systemctl restart systemd- networkd` now. They didn't show up before the reboot. This suggests that there may be issues I haven't noticed yet even with fewer than 34 peers. In our production environment not all of our peers are online all the time, so an issue affecting a few of them could go unnoticed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853956 Title: 34 wireguard peers result in invalid peer configuration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs