I'm definitely seeing inconsistent results pinging end to end across 3
hosts connected to 3 QFX switches. In some cases I can't ping end to end in
other cases I can though I also sometimes have to initiate a ping in both
directions for it to work. I honestly haven't pinned down exactly what
scenarios work and what don't. Previously I thought I had everything
working when hard coding ARP but I'm not seeing that in my tests today.
I'll try and do more testing today to narrow things down but here's a run
down of a set of testing I did a bit ago
host1 - 192.168.0.1 - d4:ae:52:c6:c9:e1
host2 - 192.166.0.2 - 54:9f:35:05:fa:2e
host3 - 192.168.0.3 - 54:9f:35:06:04:9c
contrail - 10.10.10.53
leaf1z0 - 10.10.10.65
leaf2z0 - 10.10.10.66
leaf3z0 - 10.10.10.67
Before configuring overlay ports:
=========================================================
root@leaf1z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch default 1
{master:0}
root@leaf1z0> show ethernet-switching table
{master:0}
root@leaf1z0>
root@leaf2z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch default 1
{master:0}
root@leaf2z0> show ethernet-switching table
{master:0}
root@leaf2z0>
root@leaf3z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch default 1
{master:0}
root@leaf3z0> show ethernet-switching table
{master:0}
root@leaf3z0>
After configuring ge-0/0/0 on each leaf to be in overlay network:
====================================================================
root@leaf1z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c NA
ge-0/0/0.0*
vtep.32769*
default-switch default 1
{master:0}
root@leaf1z0> show ethernet-switching table
{master:0}
root@leaf1z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..65
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
root@leaf2z0> show ethernet-switching table
{master:0}
root@leaf2z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c NA
ge-0/0/0.0*
vtep.32769*
default-switch default 1
{master:0}
root@leaf2z0> show ethernet-switching table
{master:0}
root@leaf2z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..66
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
root@leaf3z0> show vlans
Routing instance VLAN name Tag Interfaces
default-switch Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c NA
ge-0/0/0.0*
vtep.32769*
default-switch default 1
{master:0}
root@leaf3z0> show ethernet-switching table
{master:0}
root@leaf3z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..67
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
{master:0}
root@leaf3z0>
Configure end hosts with IPs and confirm initial state
====================================================================
dhoutz@host1:~$ sudo ifconfig em1 192.168.0.1/24
[sudo] password for dhoutz:
dhoutz@host1:~$ ifconfig em1
em1 Link encap:Ethernet HWaddr d4:ae:52:c6:c9:e1
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
dhoutz@host1:~$ arp -n | grep 192.168.0.
dhoutz@host1:~$
dhoutz@host2:~$ sudo ifconfig em1 192.168.0.2/24
[sudo] password for dhoutz:
dhoutz@host2:~$ ifconfig em1
em1 Link encap:Ethernet HWaddr 54:9f:35:05:fa:2e
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:16
dhoutz@host2:~$ arp -n | grep 192.168.0.
dhoutz@host2:~$
dhoutz@host3:~$ sudo ifconfig em1 192.168.0.3/24
[sudo] password for dhoutz:
dhoutz@host3:~$ ifconfig em1
em1 Link encap:Ethernet HWaddr 54:9f:35:06:04:9c
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:16
After configuring hosts, all leaf switches have learned MAC entries:
========================================================================
root@leaf1z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..65
d4:ae:52:c6:c9:e1 0.0.0.0 Vxlan over Ipv4 10.10.10..65
54:9f:35:05:fa:2e 0.0.0.0 Vxlan over Ipv4 10.10.10..66
54:9f:35:06:04:9c 0.0.0.0 Vxlan over Ipv4 10.10.10..67
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
{master:0}
root@leaf1z0> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P -
Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE
MAC, O - ovsdb MAC)
Ethernet switching table : 3 entries, 1 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:05:fa:2e
SO - vtep.32770
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:06:04:9c
SO - vtep.32771
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c d4:ae:52:c6:c9:e1
D - ge-0/0/0.0
root@leaf2z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..66
54:9f:35:05:fa:2e 0.0.0.0 Vxlan over Ipv4 10.10.10..66
54:9f:35:06:04:9c 0.0.0.0 Vxlan over Ipv4 10.10.10..67
d4:ae:52:c6:c9:e1 0.0.0.0 Vxlan over Ipv4 10.10.10..65
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
{master:0}
root@leaf2z0> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P -
Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE
MAC, O - ovsdb MAC)
Ethernet switching table : 3 entries, 1 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:05:fa:2e
D - ge-0/0/0.0
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:06:04:9c
SO - vtep.32770
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c d4:ae:52:c6:c9:e1
SO - vtep.32771
root@leaf3z0> show ovsdb mac
Logical Switch Name: Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..67
54:9f:35:06:04:9c 0.0.0.0 Vxlan over Ipv4 10.10.10..67
54:9f:35:05:fa:2e 0.0.0.0 Vxlan over Ipv4 10.10.10..66
d4:ae:52:c6:c9:e1 0.0.0.0 Vxlan over Ipv4 10.10.10..65
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.10.10..53
{master:0}
root@leaf3z0> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P -
Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE
MAC, O - ovsdb MAC)
Ethernet switching table : 3 entries, 1 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:05:fa:2e
SO - vtep.32770
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c 54:9f:35:06:04:9c
D - ge-0/0/0.0
Contrail-fbedebc0-4ed8-4f18-8309-33c71d9de71c d4:ae:52:c6:c9:e1
SO - vtep.32771
Testing host1 to host2:
================================================
dhoutz@host1:~$ ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
>From 192.168.0.1 icmp_seq=1 Destination Host Unreachable
<snip>
^C
--- 192.168.0.2 ping statistics ---
48 packets transmitted, 0 received, +45 errors, 100% packet loss, time
47254ms
host1 fails to get ARP:
dhoutz@host1:~$ arp -n | grep 192.168.0.
192.168.0.2 (incomplete)
em1
neither host2 nor host3 ever sees ARP:
dhoutz@host2:~$ sudo tcpdump -i em1 icmp or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
dhoutz@host3:~$ arp -n | grep 192.168.0
dhoutz@host3:~$ sudo tcpdump -i em1 arp or icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Testing host2 to host1:
=========================================================
dhoutz@host2:~$ ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
>From 192.168.0.2 icmp_seq=1 Destination Host Unreachable
>From 192.168.0.2 icmp_seq=2 Destination Host Unreachable
<snip>
^C
--- 192.168.0.1 ping statistics ---
26 packets transmitted, 0 received, +24 errors, 100% packet loss, time
25136ms
dhoutz@host2:~$ arp -n | grep 192.168.0.
192.168.0.1 (incomplete)
em1
Testing host2 to host3 (works!!):
==========================================================
dhoutz@host2:~$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=0.381 ms
64 bytes from 192.168.0.3: icmp_seq=4 ttl=64 time=0.333 ms
<snip>
^C
--- 192.168.0.3 ping statistics ---
14 packets transmitted, 12 received, 14% packet loss, time 13014ms
rtt min/avg/max/mdev = 0.333/0.387/0.396/0.021 ms
dhoutz@host2:~$ arp -n | grep 192.168.0.
192.168.0.3 ether 54:9f:35:06:04:9c C
em1
192.168.0.1 (incomplete)
em1
dhoutz@host3:~$ arp -n | grep 192.168.0
192.168.0.2 ether 54:9f:35:05:fa:2e C
em1
Testing host3 to host1:
===========================================================
dhoutz@host3:~$ ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
>From 192.168.0.3 icmp_seq=1 Destination Host Unreachable
>From 192.168.0.3 icmp_seq=2 Destination Host Unreachable
<snip>
^C
--- 192.168.0.1 ping statistics ---
34 packets transmitted, 0 received, +33 errors, 100% packet loss, time
33176ms
But in this case, pinging host3 -> host1 caused host1 to learn ARP:
dhoutz@host1:~$ arp -n | grep 192.168.0.
192.168.0.2 (incomplete)
em1
192.168.0.3 ether 54:9f:35:06:04:9c C
em1
However host3 still doesnt have ARP for host1:
dhoutz@host3:~$ arp -n | grep 192.168.0
192.168.0.1 (incomplete)
em1
192.168.0.2 ether 54:9f:35:05:fa:2e C
em1
On Thu, Jun 18, 2015 at 8:36 AM, Dan Houtz <dho...@gmail.com> wrote:
> Hi Harshad,
>
> This is good info and in line with what I had concluded reading through
> recent git commits. It definitely sounds like we'll need 2.2 in our
> environment as it is all bare metal and not provisioned by Openstack in any
> way - we are really just looking to use Contrail as a controller for
> hardware VETPs.
>
> I will be honest that I don't fully understand how I ever learn ARP
> entries initially if unknown unicast flooding isn't supported but I
> definitely have intermittent ARP on my end hosts.
>
> -Dan
>
> On Thu, Jun 18, 2015 at 8:02 AM, Harshad Nakil <hna...@gmail.com> wrote:
>
>> Hi Dan,
>>
>> There is problem of ARP time out in bare metal. Since we do not support
>> flooding of unknown unicast. MAC age out in QFX can happen before ARP
>> timeout in Baremetal. This will result in blackhole of traffic for that MAC.
>>
>> Work around is to set QFX MAC age out time to be larger than ARP age out
>> in Baremetal. See if this works and then you don’t have to upgrade.
>>
>> IN 2.2 we have per VN knob to support flooding for unknown unicast in
>> Baremetal case.
>>
>> Regards
>> -Harshad
>>
>> On Jun 17, 2015, at 7:17 PM, Dan Houtz <dho...@gmail.com> wrote:
>>
>> Hi Jakub,
>>
>> Thank you for the reply. To clarify a bit, we are currently running
>> Contrail 2.1 however we're seeing issues with unknown unicast for bare
>> metal servers with ARP often failing. I'm curious if we need to move to
>> something newer then 2.1 as I see a lot of recent commits that look to
>> improve this. It's all proof of concept at this point so I don't mind
>> running unrelaesed code.
>>
>> We are using:
>> QFX5100-48S running 14.1X53-D15.2.
>> MX80 running a daily build of 14.2 to get VXLAN/EVPN support.
>>
>> On Wed, Jun 17, 2015 at 1:46 AM, Jakub Pavlík <jakub.pav...@tcpcloud.eu>
>> wrote:
>>
>>> Hi Dan,
>>>
>>> if you want to deploy OpenContrail 2.1 with Juno, you need to do several
>>> steps:
>>>
>>> 1) install contrail-nova-vif together with nova-api
>>>
>>> 2) modify
>>> /usr/lib/python2.7/dist-packages/nova_contrail_vif/contrailvif.py
>>>
>>>
>>> https://raw.githubusercontent.com/pupapaik/contrail-nova-vif-driver/R2.1/nova_contrail_vif/contrailvif.py
>>>
>>> 3) After you have to set contrail api class at nova.conf
>>>
>>> network_api_class = nova_contrail_vif.contrailvif.ContrailNetworkAPI
>>>
>>> The reason is that between icehouse and juno was replaced
>>> nova_vif_driver for network_api_class.
>>>
>>> Jakub
>>> ------------------------------
>>> *Odesílate:* Dev [dev-boun...@lists.opencontrail.org] za uživatele Dan
>>> Houtz [dho...@gmail.com]
>>> *Odesláno:* 16. června 2015 21:56
>>> *Komu:* dev@lists.opencontrail.org
>>> *Předmět:* [opencontrail-dev] BMS / Unknown Unicast issues
>>>
>>> Hi,
>>>
>>> Couple questions regarding Contrail and BMS support...
>>>
>>> We are currently in the POC stages of a build-out that would be
>>> primarily bare metal based using Contrail to configure interfaces on QFX
>>> switches acting as hardware VETPs and building the proper EVPN routing
>>> instances on MX routers. The use case is to provision on demand L2 networks
>>> between bare metal dedicated servers throughout our datacenter (tied
>>> together via L3 leaf/spine underlay) for customers.
>>>
>>> Currently we have Contrail 2.1 deployed and it is properly configuring
>>> switch ports via OVSDB and seems to be distributing known macs properly
>>> between TOR switches and MX gateways. However, we are seeing a ton of
>>> issues with ARP/unknown unicast. It some cases it seems to work but not in
>>> any reliable manner.
>>>
>>> Reading through commits on github it looks like there has been a lot
>>> of work done recently on this front which leads me to a few questions:
>>>
>>> 1. Is our setup supported at all in 2.1 or will we need 2.2?
>>> 2. If 2.2 is required, does anyone have any pointers on successfully
>>> installing from master branch along with Juno? I've been able to
>>> successfully install this combo but am seeing the following errors in
>>> nova-compute logs when spinning up a VM instance:
>>> "NovaException: Unexpected vif_type=vrouter". I imagine this is due to
>>> the changing in architecture with Neutron plugins when moving to Juno. I
>>> also tried doing a devstack install with Icehouse but ran into other issues
>>> causing openstack to not even install as shown here:
>>> http://pastebin.com/5MQU6zJF
>>> 3. Ultimately I'm curious if we're on the right track and if anyone has
>>> any pointers on getting a more recent build running. Or, should we just
>>> wait for 2.2 official release assuming it's still planned for this month?
>>>
>>> Thanks!
>>> Dan H.
>>>
>>
>> _______________________________________________
>> Dev mailing list
>> Dev@lists.opencontrail.org
>> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
>>
>>
>>
>
_______________________________________________
Dev mailing list
Dev@lists.opencontrail.org
http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
