[
https://issues.apache.org/jira/browse/CLOUDSTACK-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjeev N updated CLOUDSTACK-6715:
----------------------------------
Assignee: Murali Reddy
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to
> another
> ----------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-6715
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6715
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server, Network Controller
> Affects Versions: 4.4.0
> Environment: Latest build from 4.4 with commit
> e6961fd21bb6d793302c234d0f409f66dc498072
> Reporter: Sanjeev N
> Assignee: Murali Reddy
> Priority: Critical
> Fix For: 4.4.0
>
> Attachments: management-server.rar, ovstunnel-host13.log,
> ovstunnel-host14.log
>
>
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to
> another
> Steps to reproduce:
> ================
> 1.Bring up CS in advanced zone with two xen hosts in a cluster
> 2.Create physical network with GRE isolation
> 3.Create isolated network with OVS provider
> 4.Deploy few vms in the network and make sure that all the vms(including VR)
> are deployed on only one host.
> 5.Now migrate one vm to another host in the cluster and verify flow tables
> for this isolated network bridge on both the xen hosts
> Result:
> =======
> Inconsistency in flow tables on both the xen hosts
> Following it the flow table from the host before vm migration (All the vms
> and VR were only on this host before vm migration):
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0,
> priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:1,output:2
> cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612,
> priority=0 actions=NORMAL
> cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0,
> priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:1,output:2
> cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0,
> priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494,
> priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> Flow table after vm migration on the same host:
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=NORMAL
> [root@xen-host-14 ~]#
> Ports info on the bridge after vm migration:
> [root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
> t986-2-1
> vif11.0
> vif12.0
> Flow table on the target host where vm was migrated to:
> [root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0,
> priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:2
> cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0,
> priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
> cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0,
> priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24 actions=drop
> cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=NORMAL
> cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0,
> priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:2
> cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0,
> priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
> cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42,
> priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
> [root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
> t986-1-2
> vif17.0
> Following is the log snippet from both the hosts during vm migration (tunnel
> creation during vm migration):
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
> 2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 2 to host 1
> with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
> 'xapi4', 'name']
> 2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel -
> VERIFIED
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'add-port', 'xapi4', 't986-2-1', '--', 'set', 'interface', 't986-2-1',
> 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'port', 't986-2-1', 'interfaces']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22 DEBUG [root] Tunnel interface
> validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
> '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:22 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:22 DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:22 DEBUG [root] Successfully created tunnel from host 2
> to host 1 with GRE key 986
> 2014-05-20 11:31:22 DEBUG [root] #### VMOPS exit create_tunnel ####
> 2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
> '/etc/xensource/network.conf']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'iface-to-br', 'vif19.0']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'list-ports', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 't986-2-1', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif11.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif12.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif19.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'del-flows', 'xapi4', ',in_port=3']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']
> ovstunnel log from the host where vm has been migrated to:
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter setup_ovs_bridge ####
> 2014-05-20 11:31:21 DEBUG [root] About to manually create the bridge:xapi4
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--',
> '--may-exist', 'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4',
> 'other_config:gre_key=OVSTunnel986']
> 2014-05-20 11:31:21 DEBUG [root] Bridge has been manually created:
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
> 'Bridge', 'xapi4',
> 'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
> 'Bridge', 'xapi4', 'stp_enable=true']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'bridge', 'xapi4', 'other_config:gre_key']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'other-config:is-ovs-tun-network=True']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=ovs-host-setup']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'other-config:ovs-host-setup=1,2']
> 2014-05-20 11:31:21 DEBUG [root] Setup_ovs_bridge completed with
> result:SUCCESS:xapi4
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit setup_ovs_bridge ####
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
> 2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 1 to host 2
> with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
> 'xapi4', 'name']
> 2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel -
> VERIFIED
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'add-port', 'xapi4', 't986-1-2', '--', 'set', 'interface', 't986-1-2',
> 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'port', 't986-1-2', 'interfaces']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21 DEBUG [root] Tunnel interface
> validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
> 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:21 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:21 DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:21 DEBUG [root] Successfully created tunnel from host 1
> to host 2 with GRE key 986
> 2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit create_tunnel ####
> 2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
> '/etc/xensource/network.conf']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'iface-to-br', 'vif17.0']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-list', 'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
> 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
> 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
> '--minimal']
> 2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
> (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
> 'list-ports', 'xapi4']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 't986-1-2', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
> 'Interface', 'vif17.0', 'ofport']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
> 2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
> 'add-flow', 'xapi4',
> 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
> ~
--
This message was sent by Atlassian JIRA
(v6.2#6252)
