Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
Heho, > Looking at the show nexthop output it seem bgpd does not get the RTM_IFINFO > message with the IFP_UP flag set. It still thinks the interface is down. This > is a bug > in wg(4) which probably sends the rt message before applying the flag. This makes a lot of sense; I am sadly not good enough with the codebase to supply a diff, but can test a patch if somebody writes one. With best regards, Tobias -Original Message- From: owner-m...@openbsd.org On Behalf Of Claudio Jeker Sent: Wednesday, 13 July 2022 13:13 To: Stuart Henderson Cc: misc@openbsd.org Subject: Re: OpenBGPD via (WG?) Tunnel Not Learning Routes On Wed, Jul 13, 2022 at 11:01:09AM -, Stuart Henderson wrote: > On 2022-07-13, Tobias Fiebig wrote: > > Heho, > > > > When doing what i described in my message, I get the below messages. > > > > When I set static routes, packet forwarding works fine, i.e.: > > > > gw02.dus01.as59645.net ~ # route add -inet6 2a06:d1c2::/48 > > 2a06:d1c0::dead:beef:c02 add net 2a06:d1c2::/48: gateway > > 2a06:d1c0::dead:beef:c02 > > > > bgp-test.test /etc # route add -inet6 default > > 2a06:d1c0::dead:beef:c01 add net default: gateway > > 2a06:d1c0::dead:beef:c01 > > > > Removing those routes and restarting the BGPD then also leads to a > > successful import of routes, see bgpctl sh nex at the bottom of this mail. > > > > It somehow feels like bgpd does not register that wg0 came up. > > Yes. > > You can check with "route -n monitor" that the route messages are > correctly sent when the interface is brought up, also try running bgpd > in the foreground with debug logging (bgpd -vvvd or so) and see if any > errors/warnings are logged when wg comes up. Looking at the show nexthop output it seem bgpd does not get the RTM_IFINFO message with the IFP_UP flag set. It still thinks the interface is down. This is a bug in wg(4) which probably sends the rt message before applying the flag. > > Let me try if this behavior is the same for other tunnels (eoip). > > Worth a try. Also maybe different between v4 and v6, WireGuard doesn't > really do v6 properly. The v4 part is also not great to be honest. Doing dynamic routing via WireGuard is just close to impossible with the way WireGuard is specified. It is not a simple tunnel but applies some route limits on top which you can't really disable. Also because of multicast issues you can't run ospfd over wg(4) so I had to put a gif tunnel in a wg tunnel to have dynamic routing. -- :wq Claudio
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
On Wed, Jul 13, 2022 at 11:01:09AM -, Stuart Henderson wrote: > On 2022-07-13, Tobias Fiebig wrote: > > Heho, > > > > When doing what i described in my message, I get the below messages. > > > > When I set static routes, packet forwarding works fine, i.e.: > > > > gw02.dus01.as59645.net ~ # route add -inet6 2a06:d1c2::/48 > > 2a06:d1c0::dead:beef:c02 > > add net 2a06:d1c2::/48: gateway 2a06:d1c0::dead:beef:c02 > > > > bgp-test.test /etc # route add -inet6 default 2a06:d1c0::dead:beef:c01 > > add net default: gateway 2a06:d1c0::dead:beef:c01 > > > > Removing those routes and restarting the BGPD then also leads to a > > successful import of routes, see bgpctl sh nex at the bottom of this mail. > > > > It somehow feels like bgpd does not register that wg0 came up. > > Yes. > > You can check with "route -n monitor" that the route messages are correctly > sent when the interface is brought up, also try running bgpd in the foreground > with debug logging (bgpd -vvvd or so) and see if any errors/warnings are > logged when wg comes up. Looking at the show nexthop output it seem bgpd does not get the RTM_IFINFO message with the IFP_UP flag set. It still thinks the interface is down. This is a bug in wg(4) which probably sends the rt message before applying the flag. > > Let me try if this behavior is the same for other tunnels (eoip). > > Worth a try. Also maybe different between v4 and v6, WireGuard doesn't really > do v6 properly. The v4 part is also not great to be honest. Doing dynamic routing via WireGuard is just close to impossible with the way WireGuard is specified. It is not a simple tunnel but applies some route limits on top which you can't really disable. Also because of multicast issues you can't run ospfd over wg(4) so I had to put a gif tunnel in a wg tunnel to have dynamic routing. -- :wq Claudio
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
Heho, As mentioned, I gave it a shot with eoip, and that worked as intended. What I noticed though, is that wg0 seems to stick around in bgpd, even after an ifconfig wg0 destroy; I fixed this by using another ip range for transfer and rebooting the downstream to make sure; In any case, with an eoip tunnel, things then worked. Maybe something that is sticky/not handled about wg? With best regards, Tobias ### After removing wg0 (ifconfig wg0 destroy), deconfiguring the peer, reloading bgpd, adding eoip0, and reconfig the peer bgp-test.test /etc # ifconfig wg0 wg0: no such interface bgp-test.test /etc # bgpctl sh nex Flags: * = nexthop valid Nexthop Route Prio Gateway Iface 2a06:d1c0::dead:beef:c01 2a06:d1c0::dead:beef:c01/128 3 connected wg0 (DOWN, unknown) 2a06:d1c0::dead:beef:c02 2a06:d1c0::dead:beef:c02/128 1 connected wg0 (DOWN, unknown) ### Adding an eoip tunnel; New IP range and reboot on downstream before doing 'rcctl bgpd start; mv hostname.eoip0 /etc; sh /etc/netstart eoip0; add peer to bgpd conf, bgpctl reload' bgp-test.test ~ # bgpctl sh nex Flags: * = nexthop valid Nexthop Route Prio Gateway Iface * 2a06:d1c0::dead:beef:d02 2a06:d1c0::dead:beef:d00/120132 connected eoip0 (UP, unknown)
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
On 2022-07-13, Tobias Fiebig wrote: > Heho, > > When doing what i described in my message, I get the below messages. > > When I set static routes, packet forwarding works fine, i.e.: > > gw02.dus01.as59645.net ~ # route add -inet6 2a06:d1c2::/48 > 2a06:d1c0::dead:beef:c02 > add net 2a06:d1c2::/48: gateway 2a06:d1c0::dead:beef:c02 > > bgp-test.test /etc # route add -inet6 default 2a06:d1c0::dead:beef:c01 > add net default: gateway 2a06:d1c0::dead:beef:c01 > > Removing those routes and restarting the BGPD then also leads to a successful > import of routes, see bgpctl sh nex at the bottom of this mail. > > It somehow feels like bgpd does not register that wg0 came up. Yes. You can check with "route -n monitor" that the route messages are correctly sent when the interface is brought up, also try running bgpd in the foreground with debug logging (bgpd -vvvd or so) and see if any errors/warnings are logged when wg comes up. > Let me try if this behavior is the same for other tunnels (eoip). Worth a try. Also maybe different between v4 and v6, WireGuard doesn't really do v6 properly.
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
Heho, When doing what i described in my message, I get the below messages. When I set static routes, packet forwarding works fine, i.e.: gw02.dus01.as59645.net ~ # route add -inet6 2a06:d1c2::/48 2a06:d1c0::dead:beef:c02 add net 2a06:d1c2::/48: gateway 2a06:d1c0::dead:beef:c02 bgp-test.test /etc # route add -inet6 default 2a06:d1c0::dead:beef:c01 add net default: gateway 2a06:d1c0::dead:beef:c01 Removing those routes and restarting the BGPD then also leads to a successful import of routes, see bgpctl sh nex at the bottom of this mail. It somehow feels like bgpd does not register that wg0 came up. Let me try if this behavior is the same for other tunnels (eoip). With best regards, Tobias ### Setting up wireguard interface after bgpd had been started bgp-test.test rem # bgpctl sh nex Flags: * = nexthop valid Nexthop Route Prio Gateway Iface 2a06:d1c0::dead:beef:c01 2a06:d1c0::dead:beef:c01/128 3 connected wg0 (DOWN, unknown) 2a06:d1c0::dead:beef:c02 2a06:d1c0::dead:beef:c02/128 1 connected wg0 (DOWN, unknown) bgp-test.test rem # ifconfig wg0 wg0: flags=80c3 mtu 1420 index 6 priority 0 llprio 3 wgport 13720 wgrtable 23 wgpubkey wgpeer wgpka 25 (sec) wgendpoint 2001:4ba0:92f4:3::235 2342 tx: 641944, rx: 7763244 last handshake: 33 seconds ago wgaip 0.0.0.0/0 wgaip ::/0 groups: wg inet6 2a06:d1c0::dead:beef:c02 prefixlen 120 bgp-test.test rem # bgpctl show Neighbor ASMsgRcvdMsgSent OutQ Up/Down State/PrfRcvd 2a06:d1c0::dead:beef:c0 59645 48128 12 0 00:04:06 133825 ### bgpctl sh nex after restarting bgpd bgp-test.test /etc # bgpctl sh nex Flags: * = nexthop valid Nexthop Route Prio Gateway Iface * 2a06:d1c0::dead:beef:c01 2a06:d1c0::dead:beef:c01/128 3 connected wg0 (UP, unknown) * 2a06:d1c0::dead:beef:c02 2a06:d1c0::dead:beef:c02/128 1 connected wg0 (UP, unknown) -Original Message- From: owner-m...@openbsd.org On Behalf Of Stuart Henderson Sent: Wednesday, 13 July 2022 08:14 To: misc@openbsd.org Subject: Re: OpenBGPD via (WG?) Tunnel Not Learning Routes On 2022-07-13, Tobias Fiebig wrote: > Heho, > I am running OpenBGPd (on 7.1+binpatches), and have some tunnel links between > hosts and up/downstreams over wg tunnels. > > I am basically wondering whether the behavior is known/normal and/or happened > to others, or if it is worth it to setup a test-setup to properly debug the > issue/document how it can be reproduced. > > Specifically, I noticed that bgpd will consider routes invalid which it > learns over a (wg?) interface that was not there when bgpd was started; So, > essentially: > > Start bgpd > Create wireguard interface, configure IPs Adjust bgpd config to add > new peer on that if. > bgpctl reload > > -> Session with the peer comes up, bgpd sees the routes, but it lacks the > 'valid' * flag. > > Restarting bgpd resolves this (but also lets all sessions flap). > > I did not see (or missed) something about this in the man page; The same > issue seems to not occur with other Interfaces added later, e.g., vlan. How does "bgpctl sh nex" look, both in the failed situation and the situation where wg was already created? -- Please keep replies on the mailing list.
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
On 2022-07-13, Tobias Fiebig wrote: > Heho, > I am running OpenBGPd (on 7.1+binpatches), and have some tunnel links between > hosts and up/downstreams over wg tunnels. > > I am basically wondering whether the behavior is known/normal and/or happened > to others, or if it is worth it to setup a test-setup to properly debug the > issue/document how it can be reproduced. > > Specifically, I noticed that bgpd will consider routes invalid which it > learns over a (wg?) interface that was not there when bgpd was started; So, > essentially: > > Start bgpd > Create wireguard interface, configure IPs > Adjust bgpd config to add new peer on that if. > bgpctl reload > > -> Session with the peer comes up, bgpd sees the routes, but it lacks the > 'valid' * flag. > > Restarting bgpd resolves this (but also lets all sessions flap). > > I did not see (or missed) something about this in the man page; The same > issue seems to not occur with other Interfaces added later, e.g., vlan. How does "bgpctl sh nex" look, both in the failed situation and the situation where wg was already created? -- Please keep replies on the mailing list.
Re: OpenBGPD via (WG?) Tunnel Not Learning Routes
Hello Tobias, Next hop Validation to make routes valid ? asks the question is the Next hop reachable... so if you look at the prefixes learned and the next hop... you may need additional routes to make the next hop visible (via an Interior Routing Protocol o) (OSPF RIP / EIGRP) or Static Routes ... Tip to add peering lans / Transit uplink lans to OSPF just add the network to OSPF and set the interface to passive (it is the safest way) (avoid redistribute Connected if you can) once the next hop is pingable in of its self then the routes that point to the next hop should become valid.. I hope this helps, Tom Smyth On Wed, 13 Jul 2022 at 02:38, Tobias Fiebig < tob...@reads-this-mailinglist.com> wrote: > Heho, > I am running OpenBGPd (on 7.1+binpatches), and have some tunnel links > between hosts and up/downstreams over wg tunnels. > > I am basically wondering whether the behavior is known/normal and/or > happened to others, or if it is worth it to setup a test-setup to properly > debug the issue/document how it can be reproduced. > > Specifically, I noticed that bgpd will consider routes invalid which it > learns over a (wg?) interface that was not there when bgpd was started; So, > essentially: > > Start bgpd > Create wireguard interface, configure IPs > Adjust bgpd config to add new peer on that if. > bgpctl reload > > -> Session with the peer comes up, bgpd sees the routes, but it lacks the > 'valid' * flag. > > Restarting bgpd resolves this (but also lets all sessions flap). > > I did not see (or missed) something about this in the man page; The same > issue seems to not occur with other Interfaces added later, e.g., vlan. > > With best regards, > Tobias > > > -- Kindest regards, Tom Smyth.
OpenBGPD via (WG?) Tunnel Not Learning Routes
Heho, I am running OpenBGPd (on 7.1+binpatches), and have some tunnel links between hosts and up/downstreams over wg tunnels. I am basically wondering whether the behavior is known/normal and/or happened to others, or if it is worth it to setup a test-setup to properly debug the issue/document how it can be reproduced. Specifically, I noticed that bgpd will consider routes invalid which it learns over a (wg?) interface that was not there when bgpd was started; So, essentially: Start bgpd Create wireguard interface, configure IPs Adjust bgpd config to add new peer on that if. bgpctl reload -> Session with the peer comes up, bgpd sees the routes, but it lacks the 'valid' * flag. Restarting bgpd resolves this (but also lets all sessions flap). I did not see (or missed) something about this in the man page; The same issue seems to not occur with other Interfaces added later, e.g., vlan. With best regards, Tobias
