Hi Nischal,
I'm taking a look at the virtual-gateway-address knob now. Initial testing
shows that I can transit out the virtual gateway but when I lose my first
MX my TOR does not fail over to the 2nd MX. Considering both MXs are
announcing the MAC/IP in EVPN I would think this must be a result of the
TOR agent not sending the proper updates to TORs via OVSDB. Does this sound
plausbible?
Here are details on how I am currently testing:
Config on MXs:
----------------
root@gw1z0> show configuration interfaces irb.5
family inet {
address 10.10.210.146/29 {
virtual-gateway-address 10.10.210.145;
}
}
root@gw2z0> show configuration interfaces irb.5
family inet {
address 10.10.210.147/29 {
virtual-gateway-address 10.10.210.145;
}
}
EVPN Database on MXs under normal circumstances:
-------------------------------------------------
root@gw1z0> show evpn database
Instance: Demo-NetworkA
VLAN VNI MAC address Active source
Timestamp IP address
4 00:00:5e:00:01:01 05:00:00:ff:84:00:00:00:04:00 Jun 25
19:14:25 10.10.210.145
4 50:c5:8d:d4:af:f0 10.10.214.71 Jun 25
19:14:25 10.10.210.147
4 54:9f:35:05:fa:2e 10.10.214.66 Jun 25 19:14:24
4 f8:c0:01:19:b1:88 irb.5 Jun 25
19:13:46 10.10.210.146
root@gw2z0> show evpn database
Instance: Demo-NetworkA
VLAN VNI MAC address Active source
Timestamp IP address
4 00:00:5e:00:01:01 05:00:00:ff:84:00:00:00:04:00 Jun 25
17:43:27 10.10.210.145
4 50:c5:8d:d4:af:f0 irb.5 Jun 25
17:16:46 10.10.210.147
4 54:9f:35:05:fa:2e 10.10.214.66 Jun 25 17:17:15
4 f8:c0:01:19:b1:88 10.10.214.70 Jun 25
17:43:27 10.10.210.146
EVPN advertisements on MXs:
----------------------------
root@gw1z0> show route advertising-protocol bgp 10.10.210.139
__contrail__default-domain_admin_Network1.inet.0: 1 destinations, 2 routes
(1 active, 0 holddown, 1 hidden)
Prefix Nexthop MED Lclpref AS path
* 192.168.1.0/24 Self 100 I
Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0
hidden)
Prefix Nexthop MED Lclpref AS path
2:10.10.214.70:2::4::00:00:5e:00:01:01/304
* Self 100 I
2:10.10.214.70:2::4::f8:c0:01:19:b1:88/304
* Self 100 I
2:10.10.214.70:2::4::00:00:5e:00:01:01::10.10.210.145/304
* Self 100 I
2:10.10.214.70:2::4::f8:c0:01:19:b1:88::10.10.210.146/304
* Self 100 I
3:10.10.214.70:2::4::10.10.214.70/304
* Self 100 I
__default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
hidden)
Prefix Nexthop MED Lclpref AS path
1:10.10.214.70:0::050000ff840000000400::FFFF:FFFF/304
* Self 100 I
root@gw2z0> show route advertising-protocol bgp 10.10.210.139
__contrail__default-domain_admin_Network1.inet.0: 1 destinations, 2 routes
(1 active, 0 holddown, 1 hidden)
Prefix Nexthop MED Lclpref AS path
* 192.168.1.0/24 Self 100 I
Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0
hidden)
Prefix Nexthop MED Lclpref AS path
2:10.10.214.71:1::4::00:00:5e:00:01:01/304
* Self 100 I
2:10.10.214.71:1::4::50:c5:8d:d4:af:f0/304
* Self 100 I
2:10.10.214.71:1::4::00:00:5e:00:01:01::10.10.210.145/304
* Self 100 I
2:10.10.214.71:1::4::50:c5:8d:d4:af:f0::10.10.210.147/304
* Self 100 I
3:10.10.214.71:1::4::10.10.214.71/304
* Self 100 I
__default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
hidden)
Prefix Nexthop MED Lclpref AS path
1:10.10.214.71:0::050000ff840000000400::FFFF:FFFF/304
* Self 100 I
With both MXs up, my host can transit the gateway:
------------------------------------------------------
dhoutz@host2:/$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=1.38 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=1.34 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.346/1.364/1.382/0.018 ms
dhoutz@host2:/$ netstat -rn | grep 0.0.0.0
0.0.0.0 10.10.210.145 0.0.0.0 UG 0 0 0 em1
And my TOR shows a VTEP destination of MX1 (10.10.214.70):
-------------------------------------------------------------
root@leaf1z0> show ovsdb mac | grep 00:00:5e:00:01:01
00:00:5e:00:01:01 0.0.0.0 Vxlan over Ipv4 10.10.214.70
If I shutdown MX1 my TOR never updates updates OVSDB and remains the same
as above.
Thanks!
Dan
On Thu, Jun 25, 2015 at 11:33 AM, Nischal Sheth <[email protected]> wrote:
>
> Dan and Alex,
>
> A slight variation of the below is the recommended configuration.
> Instead of
> configuring the same IP on the MX IRBs, you would configure unique IPs on
> the IRBs so that connectivity to an individual MX can be tested. A new knob
> called virtual-gateway-address is added under the irb.xxx IP address. You
> would set the this to the subnet's gateway address on both MXs. The GW IP
> will be advertised with the same VRRP MAC by both MXs.
>
> Robert, this should work even with local VNI in future because the VNI is
> determined by doing a lookup of the MAC address on the TOR switch.
> Same if the encapsulation is MPLS instead of VXLAN.
>
> -Nischal
>
> On Jun 25, 2015, at 9:23 AM, Dan Houtz <[email protected]> wrote:
>
> Hi Alex,
>
> I agree that anycastting the gateway would be ideal. I have tested this a
> bit but haven't had any luck thus fa as my TOR's only ever show one path.
> Even if I power down the MX that my TOR's have as the VTEP destination for
> the hard coded MAC, the entry is never removed from the OVSDB / mac table.
>
> Going to test this a bit more and see if I can find anything. I'll
> provide some results shortly.
>
> Thanks!
> Dan
>
>
>
>
> On Thu, Jun 25, 2015 at 3:56 AM, Alex Walker <[email protected]
> > wrote:
>
>> Hi Dan,
>>
>>
>>
>> Had you considered not using VRRP, and configuring both of your MX IRBs
>> to use the same IP address and MAC? You’ll need to think about it for your
>> own situation, but this could achieve the redundancy you require?
>>
>>
>>
>> Each MX will announce the same MAC via EVPN BGP. If either next-hop is
>> used, either should forward traffic OK for you. Should one gateway fail,
>> standard BGP path selection will then simply select the other gateway. You
>> could probably even play with Local Preference to prefer one gateway over
>> the other?
>>
>>
>>
>> Hope this helps – good luck with it!
>>
>>
>>
>> Alex.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Users [mailto:[email protected]] *On Behalf
>> Of *Dan Houtz
>> *Sent:* 25 June 2015 05:31
>> *To:* [email protected]
>> *Subject:* [Users] Redundant L3 gateways using MX80s
>>
>>
>>
>> Hi,
>>
>>
>>
>> I've been working hard to get BMS / TOR switch VXLAN overlays working
>> and, with the gracious help of Praveen, I believe everything is mostly
>> working as expected now. With that functionality in a working state I'm now
>> looking at adding in L3 gateway support using a pair of MX80s. This seems
>> pretty straight forward with one MX but I haven't quite got a redundant
>> setup working properly. Are there any documented 'best practices' for
>> this? In an attempt to add redundancy I am trying to configure VRRP (I'm
>> not sure EVPN anycast gateways are viable in this setup. Hopefully someone
>> can confirm).
>>
>>
>>
>> At present VRRP does converge across the overlay, my hosts can ping the
>> physical IP addresses configured on each MX IRB, and my hosts succeed in
>> getting ARP for the VRRP VIPs. They can not, however, complete a ping (I do
>> have accept-data configured for VRRP so MX should reply to pings). After a
>> bit of digging I believe this is because the virtual MAC address of the VIP
>> is not being announced via BGP so my TOR switches never learn the MAC via
>> OVSDB.
>>
>>
>>
>> root@gw1z0> show configuration interfaces irb.5
>>
>> family inet {
>>
>> address 10.10.10.146/29 {
>>
>> vrrp-group 13 {
>>
>> virtual-address 10.10.10.145;
>>
>> priority 105;
>>
>> accept-data;
>>
>> }
>>
>> }
>>
>> }
>>
>>
>>
>> root@gw1z0> show route advertising-protocol bgp 10.10.10.139
>>
>>
>>
>> Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown,
>> 0 hidden)
>>
>> Prefix Nexthop MED Lclpref AS path
>>
>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88/304
>>
>> * Self 100 I
>>
>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88::10.10.10.145/304
>>
>> * Self 100 I
>>
>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88::10.10.10.146/304
>>
>> * Self 100 I
>>
>> 3:10.10.20.70:2::4::10.10.20.70/304
>>
>> * Self 100 I
>>
>>
>>
>> __default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown,
>> 0 hidden)
>>
>> Prefix Nexthop MED Lclpref AS path
>>
>> 1:10.10.20.70:0::050000ff840000000400::FFFF:FFFF/304
>>
>> * Self 100 I
>>
>>
>>
>> As you can see in the advertised BGP routes above the MX is announcing
>> .145 with the actual MAC of the IRB:
>>
>>
>>
>> root@gw1z0> show interfaces irb | grep hardware
>>
>> Current address: f8:c0:01:19:b1:88, Hardware address: f8:c0:01:19:b1:88
>>
>>
>>
>> not the virtual MAC that has been assigned for VRRP:
>>
>>
>>
>> root@gw1z0> show vrrp extensive | grep mac
>>
>> Virtual Mac: 00:00:5e:00:01:0d
>>
>>
>>
>> I suspect that because the MX never announces an EVPN mac of
>> 00:00:5e:00:01:0d, my TOR never learns where that MAC lives:
>>
>>
>>
>> oot@leaf1z0> show ovsdb mac
>>
>> Logical Switch Name: Contrail-12905c4b-60db-4d71-8fab-972051db1386
>>
>> Mac IP Encapsulation Vtep
>>
>> Address Address Address
>>
>> ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.20.65
>>
>> d4:ae:52:c6:c9:e1 0.0.0.0 Vxlan over Ipv4 10.10.20.65
>>
>> 50:c5:8d:d4:af:f0 0.0.0.0 Vxlan over Ipv4 10.10.20.71
>>
>> 54:9f:35:05:fa:2e 0.0.0.0 Vxlan over Ipv4 10.10.20.66
>>
>> f8:c0:01:19:b1:88 0.0.0.0 Vxlan over Ipv4 10.10.20.70
>>
>> ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4
>> 10.10.10.139
>>
>>
>>
>>
>>
>> Is there perhaps a knob I'm missing? Is there something other then VRRP
>> that is preferred for redundant gateways in Contrail?
>>
>>
>>
>> I have the same question out to a few other resources but figured someone
>> on the mailing list might have some good pointers as well.
>>
>>
>>
>> Thanks!
>>
>> Dan
>>
>
> _______________________________________________
> Users mailing list
> [email protected]
> http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
>
>
>
_______________________________________________
Users mailing list
[email protected]
http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
