On Fri, Feb 15, 2019 at 01:12:36PM +1100, Adrian Close wrote:
> >Synopsis: Incorrect ARPing for routed LDP peer's loopback IP instead of
> ARPing for next-hop router IP on expiry
> >Category: system
> >Environment:
> System : OpenBSD 6.4
> Details : OpenBSD 6.4-current (GENERIC) #688: Wed Feb 13 12:16:06 MST
> 2019
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
>
> Architecture: OpenBSD.amd64
> Machine : amd64
> >Description:
> I'm having some issues getting an mpw(4) MPLS "pseudowire" setup working
> reliably and it seems to come down to ARP issues between my "PE" and "P"
> boxes that occur when ldpd(8) thinks the pseudowire l2vpn tunnels are
> actually up.
>
> My setup looks something like:
>
> [cust L2] - [vic0]PE1[vic1] -- [vic0]P1[vic1] -- [vic1]P2[vic0] --
> [vic1][PE2][vic0] - [cust L2]
>
> ... the idea being to provide a layer 2 circuit between the "cust L2"
> ports, over an MPLS network, which I've configured along the lines of
> Claudio's "Demystifying MPLS" paper and Renato's "VPLS basic test
> setup", using LDP and OSPF. This all works fine, except when it doesn't:
>
> So for example, when the ARP entry on the PE1 box for P1's vic0 IP
> address times out (or is manually deleted), PE1 sends out ARP requests
> not for that IP address, but for the loopback address of its pseudowire
> peer PE2.
>
> It generally does this for about 40 seconds, and then finally sends an
> ARP request for the correct IP, which is answered straight away and
> things work again until next time.
>
> If ldpd(8) thinks the l2vpn tunnels are down ("ldpctl show l2vpn
> pseudowires"), ARP requests are sent normally and work as expected.
>
>
> >How-To-Repeat:
> I made four VMs in VMWare ESX and connected them together in series,
> ie. PE1[vic1]:[vic0]P1[vic1]:[vic1]P2[vic0]:[vic1]PE2.
> Configure networking and osfpd/lpd as follows:
>
> PE1 configuration:
> ospfd.conf:
> router-id 172.31.211.101
> area 0.0.0.0 {
> interface vic1
> interface lo1
> }
> ldpd.conf:
> router-id 172.31.211.101
> address-family ipv4 {
> interface vic1
> }
> l2vpn VMVPLS type vpls {
> bridge bridge0
> interface vic0
> pseudowire mpw2 {
> pw-id 102
> neighbor-id 172.31.221.101
> }
> }
> vic1: 192.168.11.101/24, MPLS enabled
> lo1: 172.31.211.101/32
> bridge0: members vic0 + mpw2
>
> P1 configuration:
> ospfd.conf:
> router-id 172.31.100.11
> area 0.0.0.0 {
> interface vic0
> interface vic1
> interface lo1
> }
> ldpd.conf:
> router-id 172.31.100.11
> address-family ipv4 {
> interface vic0
> interface vic1
> }
> vic0: 192.168.11.11/24, MPLS enabled
> vic1: 192.168.12.11/24, MPLS enabled
> lo1: 172.31.100.11/32
>
> P2 configuration:
> ospfd.conf:
> router-id 172.31.100.21
> area 0.0.0.0 {
> interface vic0
> interface vic1
> interface lo1
> }
> ldpd.conf:
> router-id 172.31.100.21
> address-family ipv4 {
> interface vic0
> interface vic1
> }
> vic0: 192.168.12.21/24, MPLS enabled
> vic1: 192.168.22.21/24, MPLS enabled
> lo1: 172.31.100.21/32
>
> PE2 configuration:
> ospfd.conf:
> router-id 172.31.221.101
> area 0.0.0.0 {
> interface vic1
> interface lo1
> }
> ldpd.conf:
> router-id 172.31.221.101
> address-family ipv4 {
> interface vic1
> }
> l2vpn VMVPLS type vpls {
> bridge bridge0
> interface vic0
> pseudowire mpw2 {
> pw-id 102
> neighbor-id 172.31.211.101
> }
> }
> vic1: 192.168.22.201/24, MPLS enabled
> lo1: 172.31.211.101/32
> bridge0: members vic0 + mpw2
>
>
> Bring up ospfd and ldpd on each box. Routing and MPLS should converge OK.
> The L2VPN should come up (see 'ldpctl show l2vpn pseudowires' output).
> Ethernet frames pass between vic0 interfaces on the PE boxes, via MPLS.
>
> Once the ARP entry on PE1 for P1 (192.168.11.11) and/or on PE2 for P2
> (192.168.22.21) times out, the pseudowire stops working. Similar behaviour
> if
> the ARP entry is manually deleted.
>
> The PE box will then start ARPing on vic1 for its pseudowire peer
> (eg. 172.31.211.101) instead of its nexthop router for that peer, which of
> course doesn't work. After a while it does ARP for the nexthop router
> (eg. 192.168.11.11), P responds and connectivity is restored.
>
>
> >Fix:
> Configure static ARP entries for P router IP on PE boxes to work
> around this. Ew.
Hi Adrian,
Could you please try the following diff? I was able to reproduce the
problem you were seeing, but this makes it go away.
Index: if_ethersubr.c
===================================================================
RCS file: /cvs/src/sys/net/if_ethersubr.c,v
retrieving revision 1.258
diff -u -p -r1.258 if_ethersubr.c
--- if_ethersubr.c 18 Feb 2019 03:41:21 -0000 1.258
+++ if_ethersubr.c 19 Feb 2019 03:23:15 -0000
@@ -235,19 +235,16 @@ ether_resolve(struct ifnet *ifp, struct
#endif
#ifdef MPLS
case AF_MPLS:
- if (rt)
- dst = rt_key(rt);
- else
+ if (rt == NULL)
senderr(EHOSTUNREACH);
if (!ISSET(ifp->if_xflags, IFXF_MPLS))
senderr(ENETUNREACH);
- af = dst->sa_family;
- if (af == AF_MPLS)
- af = rt->rt_gateway->sa_family;
+ dst = ISSET(rt->rt_flags, RTF_GATEWAY) ?
+ rt->rt_gateway : rt_key(rt);
- switch (af) {
+ switch (dst->sa_family) {
case AF_LINK:
if (satosdl(dst)->sdl_alen < sizeof(eh->ether_dhost))
senderr(EHOSTUNREACH);
