I should also mention that I am currently running 2.1 release of contrail.
On Jun 25, 2015 12:58 PM, "Dan Houtz" <[email protected]> wrote:
> Hi Nischal,
>
> I'm taking a look at the virtual-gateway-address knob now. Initial testing
> shows that I can transit out the virtual gateway but when I lose my first
> MX my TOR does not fail over to the 2nd MX. Considering both MXs are
> announcing the MAC/IP in EVPN I would think this must be a result of the
> TOR agent not sending the proper updates to TORs via OVSDB. Does this sound
> plausbible?
>
> Here are details on how I am currently testing:
>
> Config on MXs:
> ----------------
>
> root@gw1z0> show configuration interfaces irb.5
> family inet {
> address 10.10.210.146/29 {
> virtual-gateway-address 10.10.210.145;
> }
> }
>
> root@gw2z0> show configuration interfaces irb.5
> family inet {
> address 10.10.210.147/29 {
> virtual-gateway-address 10.10.210.145;
> }
> }
>
>
> EVPN Database on MXs under normal circumstances:
> -------------------------------------------------
>
> root@gw1z0> show evpn database
> Instance: Demo-NetworkA
> VLAN VNI MAC address Active source
> Timestamp IP address
> 4 00:00:5e:00:01:01 05:00:00:ff:84:00:00:00:04:00 Jun 25
> 19:14:25 10.10.210.145
> 4 50:c5:8d:d4:af:f0 10.10.214.71 Jun 25
> 19:14:25 10.10.210.147
> 4 54:9f:35:05:fa:2e 10.10.214.66 Jun 25 19:14:24
> 4 f8:c0:01:19:b1:88 irb.5 Jun 25
> 19:13:46 10.10.210.146
>
>
> root@gw2z0> show evpn database
> Instance: Demo-NetworkA
> VLAN VNI MAC address Active source
> Timestamp IP address
> 4 00:00:5e:00:01:01 05:00:00:ff:84:00:00:00:04:00 Jun 25
> 17:43:27 10.10.210.145
> 4 50:c5:8d:d4:af:f0 irb.5 Jun 25
> 17:16:46 10.10.210.147
> 4 54:9f:35:05:fa:2e 10.10.214.66 Jun 25 17:17:15
> 4 f8:c0:01:19:b1:88 10.10.214.70 Jun 25
> 17:43:27 10.10.210.146
>
>
> EVPN advertisements on MXs:
> ----------------------------
>
> root@gw1z0> show route advertising-protocol bgp 10.10.210.139
>
> __contrail__default-domain_admin_Network1.inet.0: 1 destinations, 2 routes
> (1 active, 0 holddown, 1 hidden)
> Prefix Nexthop MED Lclpref AS path
> * 192.168.1.0/24 Self 100 I
>
> Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0
> hidden)
> Prefix Nexthop MED Lclpref AS path
> 2:10.10.214.70:2::4::00:00:5e:00:01:01/304
> * Self 100 I
> 2:10.10.214.70:2::4::f8:c0:01:19:b1:88/304
> * Self 100 I
> 2:10.10.214.70:2::4::00:00:5e:00:01:01::10.10.210.145/304
> * Self 100 I
> 2:10.10.214.70:2::4::f8:c0:01:19:b1:88::10.10.210.146/304
> * Self 100 I
> 3:10.10.214.70:2::4::10.10.214.70/304
> * Self 100 I
>
> __default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
> hidden)
> Prefix Nexthop MED Lclpref AS path
> 1:10.10.214.70:0::050000ff840000000400::FFFF:FFFF/304
> * Self 100 I
>
>
> root@gw2z0> show route advertising-protocol bgp 10.10.210.139
>
> __contrail__default-domain_admin_Network1.inet.0: 1 destinations, 2 routes
> (1 active, 0 holddown, 1 hidden)
> Prefix Nexthop MED Lclpref AS path
> * 192.168.1.0/24 Self 100 I
>
> Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0
> hidden)
> Prefix Nexthop MED Lclpref AS path
> 2:10.10.214.71:1::4::00:00:5e:00:01:01/304
> * Self 100 I
> 2:10.10.214.71:1::4::50:c5:8d:d4:af:f0/304
> * Self 100 I
> 2:10.10.214.71:1::4::00:00:5e:00:01:01::10.10.210.145/304
> * Self 100 I
> 2:10.10.214.71:1::4::50:c5:8d:d4:af:f0::10.10.210.147/304
> * Self 100 I
> 3:10.10.214.71:1::4::10.10.214.71/304
> * Self 100 I
>
> __default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
> hidden)
> Prefix Nexthop MED Lclpref AS path
> 1:10.10.214.71:0::050000ff840000000400::FFFF:FFFF/304
> * Self 100 I
>
>
> With both MXs up, my host can transit the gateway:
> ------------------------------------------------------
>
> dhoutz@host2:/$ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> 64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=1.38 ms
> 64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=1.34 ms
> ^C
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> rtt min/avg/max/mdev = 1.346/1.364/1.382/0.018 ms
>
>
> dhoutz@host2:/$ netstat -rn | grep 0.0.0.0
> 0.0.0.0 10.10.210.145 0.0.0.0 UG 0 0 0 em1
>
>
> And my TOR shows a VTEP destination of MX1 (10.10.214.70):
> -------------------------------------------------------------
> root@leaf1z0> show ovsdb mac | grep 00:00:5e:00:01:01
> 00:00:5e:00:01:01 0.0.0.0 Vxlan over Ipv4 10.10.214.70
>
>
> If I shutdown MX1 my TOR never updates updates OVSDB and remains the same
> as above.
>
> Thanks!
> Dan
>
> On Thu, Jun 25, 2015 at 11:33 AM, Nischal Sheth <[email protected]>
> wrote:
>
>>
>> Dan and Alex,
>>
>> A slight variation of the below is the recommended configuration.
>> Instead of
>> configuring the same IP on the MX IRBs, you would configure unique IPs on
>> the IRBs so that connectivity to an individual MX can be tested. A new
>> knob
>> called virtual-gateway-address is added under the irb.xxx IP address. You
>> would set the this to the subnet's gateway address on both MXs. The GW IP
>> will be advertised with the same VRRP MAC by both MXs.
>>
>> Robert, this should work even with local VNI in future because the VNI
>> is
>> determined by doing a lookup of the MAC address on the TOR switch.
>> Same if the encapsulation is MPLS instead of VXLAN.
>>
>> -Nischal
>>
>> On Jun 25, 2015, at 9:23 AM, Dan Houtz <[email protected]> wrote:
>>
>> Hi Alex,
>>
>> I agree that anycastting the gateway would be ideal. I have tested this a
>> bit but haven't had any luck thus fa as my TOR's only ever show one path.
>> Even if I power down the MX that my TOR's have as the VTEP destination for
>> the hard coded MAC, the entry is never removed from the OVSDB / mac table.
>>
>> Going to test this a bit more and see if I can find anything. I'll
>> provide some results shortly.
>>
>> Thanks!
>> Dan
>>
>>
>>
>>
>> On Thu, Jun 25, 2015 at 3:56 AM, Alex Walker <
>> [email protected]> wrote:
>>
>>> Hi Dan,
>>>
>>>
>>>
>>> Had you considered not using VRRP, and configuring both of your MX IRBs
>>> to use the same IP address and MAC? You’ll need to think about it for your
>>> own situation, but this could achieve the redundancy you require?
>>>
>>>
>>>
>>> Each MX will announce the same MAC via EVPN BGP. If either next-hop is
>>> used, either should forward traffic OK for you. Should one gateway fail,
>>> standard BGP path selection will then simply select the other gateway. You
>>> could probably even play with Local Preference to prefer one gateway over
>>> the other?
>>>
>>>
>>>
>>> Hope this helps – good luck with it!
>>>
>>>
>>>
>>> Alex.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Users [mailto:[email protected]] *On Behalf
>>> Of *Dan Houtz
>>> *Sent:* 25 June 2015 05:31
>>> *To:* [email protected]
>>> *Subject:* [Users] Redundant L3 gateways using MX80s
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I've been working hard to get BMS / TOR switch VXLAN overlays working
>>> and, with the gracious help of Praveen, I believe everything is mostly
>>> working as expected now. With that functionality in a working state I'm now
>>> looking at adding in L3 gateway support using a pair of MX80s. This seems
>>> pretty straight forward with one MX but I haven't quite got a redundant
>>> setup working properly. Are there any documented 'best practices' for
>>> this? In an attempt to add redundancy I am trying to configure VRRP (I'm
>>> not sure EVPN anycast gateways are viable in this setup. Hopefully someone
>>> can confirm).
>>>
>>>
>>>
>>> At present VRRP does converge across the overlay, my hosts can ping the
>>> physical IP addresses configured on each MX IRB, and my hosts succeed in
>>> getting ARP for the VRRP VIPs. They can not, however, complete a ping (I do
>>> have accept-data configured for VRRP so MX should reply to pings). After a
>>> bit of digging I believe this is because the virtual MAC address of the VIP
>>> is not being announced via BGP so my TOR switches never learn the MAC via
>>> OVSDB.
>>>
>>>
>>>
>>> root@gw1z0> show configuration interfaces irb.5
>>>
>>> family inet {
>>>
>>> address 10.10.10.146/29 {
>>>
>>> vrrp-group 13 {
>>>
>>> virtual-address 10.10.10.145;
>>>
>>> priority 105;
>>>
>>> accept-data;
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> root@gw1z0> show route advertising-protocol bgp 10.10.10.139
>>>
>>>
>>>
>>> Demo-NetworkA.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown,
>>> 0 hidden)
>>>
>>> Prefix Nexthop MED Lclpref AS path
>>>
>>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88/304
>>>
>>> * Self 100 I
>>>
>>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88::10.10.10.145/304
>>>
>>> * Self 100 I
>>>
>>> 2:10.10.20.70:2::4::f8:c0:01:19:b1:88::10.10.10.146/304
>>>
>>> * Self 100 I
>>>
>>> 3:10.10.20.70:2::4::10.10.20.70/304
>>>
>>> * Self 100 I
>>>
>>>
>>>
>>> __default_evpn__.evpn.0: 1 destinations, 1 routes (1 active, 0 holddown,
>>> 0 hidden)
>>>
>>> Prefix Nexthop MED Lclpref AS path
>>>
>>> 1:10.10.20.70:0::050000ff840000000400::FFFF:FFFF/304
>>>
>>> * Self 100 I
>>>
>>>
>>>
>>> As you can see in the advertised BGP routes above the MX is announcing
>>> .145 with the actual MAC of the IRB:
>>>
>>>
>>>
>>> root@gw1z0> show interfaces irb | grep hardware
>>>
>>> Current address: f8:c0:01:19:b1:88, Hardware address: f8:c0:01:19:b1:88
>>>
>>>
>>>
>>> not the virtual MAC that has been assigned for VRRP:
>>>
>>>
>>>
>>> root@gw1z0> show vrrp extensive | grep mac
>>>
>>> Virtual Mac: 00:00:5e:00:01:0d
>>>
>>>
>>>
>>> I suspect that because the MX never announces an EVPN mac of
>>> 00:00:5e:00:01:0d, my TOR never learns where that MAC lives:
>>>
>>>
>>>
>>> oot@leaf1z0> show ovsdb mac
>>>
>>> Logical Switch Name: Contrail-12905c4b-60db-4d71-8fab-972051db1386
>>>
>>> Mac IP Encapsulation Vtep
>>>
>>> Address Address Address
>>>
>>> ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4
>>> 10.10.20.65
>>>
>>> d4:ae:52:c6:c9:e1 0.0.0.0 Vxlan over Ipv4
>>> 10.10.20.65
>>>
>>> 50:c5:8d:d4:af:f0 0.0.0.0 Vxlan over Ipv4
>>> 10.10.20.71
>>>
>>> 54:9f:35:05:fa:2e 0.0.0.0 Vxlan over Ipv4
>>> 10.10.20.66
>>>
>>> f8:c0:01:19:b1:88 0.0.0.0 Vxlan over Ipv4
>>> 10.10.20.70
>>>
>>> ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4
>>> 10.10.10.139
>>>
>>>
>>>
>>>
>>>
>>> Is there perhaps a knob I'm missing? Is there something other then VRRP
>>> that is preferred for redundant gateways in Contrail?
>>>
>>>
>>>
>>> I have the same question out to a few other resources but figured
>>> someone on the mailing list might have some good pointers as well.
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Dan
>>>
>>
>> _______________________________________________
>> Users mailing list
>> [email protected]
>>
>> http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
>>
>>
>>
>
_______________________________________________
Users mailing list
[email protected]
http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
