[
https://issues.apache.org/jira/browse/CLOUDSTACK-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murali Reddy resolved CLOUDSTACK-6715.
--------------------------------------
Resolution: Duplicate
Root cause is same as 6779. There is crash of OVSswitch daemon which results in
bridges getting recreated. Once bridge is recreated we loose all the flow rules
configured for the brdige.
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to
> another
> ----------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-6715
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6715
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server, Network Controller
> Affects Versions: 4.4.0
> Environment: Latest build from 4.4 with commit
> e6961fd21bb6d793302c234d0f409f66dc498072
> Reporter: Sanjeev N
> Assignee: Murali Reddy
> Priority: Critical
> Fix For: 4.4.0
>
> Attachments: management-server.rar, ovstunnel-host13.log,
> ovstunnel-host14.log
>
>
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to
> another
> Steps to reproduce:
> ================
> 1.Bring up CS in advanced zone with two xen hosts in a cluster
> 2.Create physical network with GRE isolation
> 3.Create isolated network with OVS provider
> 4.Deploy few vms in the network and make sure that all the vms(including VR)
> are deployed on only one host.
> 5.Now migrate one vm to another host in the cluster and verify flow tables
> for this isolated network bridge on both the xen hosts
> Result:
> =======
> Inconsistency in flow tables on both the xen hosts
> Following it the flow table from the host before vm migration (All the vms
> and VR were only on this host before vm migration):
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0,
> priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:1,output:2
> cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612,
> priority=0 actions=NORMAL
> cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0,
> priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:1,output:2
> cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0,
> priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494,
> priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> Flow table after vm migration on the same host:
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=NORMAL
> [root@xen-host-14 ~]#
> Ports info on the bridge after vm migration:
> [root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
> t986-2-1
> vif11.0
> vif12.0
> Flow table on the target host where vm was migrated to:
> [root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0,
> priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:2
> cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0,
> priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24 actions=drop
> cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=NORMAL
> cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0,
> priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:2
> cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0,
> priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
> cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42,
> priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> [root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
> t986-1-2
> vif17.0
> Following is the log snippet from both the hosts during vm migration (tunnel
> creation during vm migration):
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
> 2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 2 to host 1
> with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
> 'xapi4', 'name']
> 2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel -
> VERIFIED
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'add-port', 'xapi4', 't986-2-1', '--', 'set', 'interface', 't986-2-1',
> 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'port', 't986-2-1', 'interfaces']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22 DEBUG [root] Tunnel interface
> validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
> '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:22 DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:22 DEBUG [root] Successfully created tunnel from host 2
> to host 1 with GRE key 986
> 2014-05-20 11:31:22 DEBUG [root] #### VMOPS exit create_tunnel ####
> 2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
> '/etc/xensource/network.conf']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'iface-to-br', 'vif19.0']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'list-ports', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 't986-2-1', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif11.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif12.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif19.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'del-flows', 'xapi4', ',in_port=3']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']
> ovstunnel log from the host where vm has been migrated to:
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter setup_ovs_bridge ####
> 2014-05-20 11:31:21 DEBUG [root] About to manually create the bridge:xapi4
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--',
> '--may-exist', 'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4',
> 'other_config:gre_key=OVSTunnel986']
> 2014-05-20 11:31:21 DEBUG [root] Bridge has been manually created:
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
> 'Bridge', 'xapi4',
> 'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
> 'Bridge', 'xapi4', 'stp_enable=true']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'bridge', 'xapi4', 'other_config:gre_key']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'other-config:is-ovs-tun-network=True']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=ovs-host-setup']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'other-config:ovs-host-setup=1,2']
> 2014-05-20 11:31:21 DEBUG [root] Setup_ovs_bridge completed with
> result:SUCCESS:xapi4
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit setup_ovs_bridge ####
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
> 2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 1 to host 2
> with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
> 'xapi4', 'name']
> 2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel -
> VERIFIED
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'add-port', 'xapi4', 't986-1-2', '--', 'set', 'interface', 't986-1-2',
> 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'port', 't986-1-2', 'interfaces']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21 DEBUG [root] Tunnel interface
> validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
> 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:21 DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:21 DEBUG [root] Successfully created tunnel from host 1
> to host 2 with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit create_tunnel ####
> 2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
> '/etc/xensource/network.conf']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'iface-to-br', 'vif17.0']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'list-ports', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 't986-1-2', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif17.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
> ~
--
This message was sent by Atlassian JIRA
(v6.2#6252)
