Public bug reported:
Description
===========
Wrong representor port was unplugged from OVS during cold migration.
This happens when VM is scheduled to use a different PCI device to target host
vs.
to what PCI device it is using from source host. Nova uses new PCI device
information to unplug
representor port in source compute.
Steps to reproduce
==================
1. Create representor ports
$ openstack port create --network private --vnic-type=direct --binding-profile
'{"capabilities": ["switchdev"]}' direct_port1
$ openstack port create --network private --vnic-type=direct --binding-profile
'{"capabilities": ["switchdev"]}' direct_port2
2. Create VMs using the ports created above:
openstack server create --flavor m1.small --image fedora24 --nic
port-id=direct_port1 --availability-zone=nova:compute-1 vm1
openstack server create --flavor m1.small --image fedora24 --nic
port-id=direct_port2 --availability-zone=nova:compute-2 vm2
3. Migrate VM2
$ openstack server migrate vm2
$ openstack server resize --confirm vm2
4. VM2 was migrated to compute-1, however representor port is still attached to
OVS
$ sudo ovs-dpctl show
system@ovs-system:
lookups: hit:466465 missed:5411 lost:0
flows: 12
masks: hit:739146 total:2 hit/pkt:1.57
port 0: ovs-system (internal)
port 1: br-pro0.0 (internal)
port 2: br-pro0 (internal)
port 3: ens6f0
port 4: br-int (internal)
port 5: eth3
Expected result
===============
After cold migration, VM's previously used representor port should be unplugged
from OVS
Actual result
=============
VM's previously used representor port is still plugged in source host. In some
scenarios, wrong representor port was unplugged from source host. Thus
affecting VMs that were not cold migrated.
Environment
===========
Libvirt+KVM
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0
$ virsh --version
3.9.0
Neutron+OVS HW Offload
Openstack Queens openstack-nova-compute-17.0.7-1
Logs & Configs
==============
1. Plug vif device using pci address 0000:81:00.5
2018-12-15 13:12:04.871 108055 DEBUG os_vif
[req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif
VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76
2. VM was migrated from compute-1 to compute-2. New pci device is now
0000:81:00.4
2018-12-15 13:13:58.721 108055 DEBUG os_vif
[req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif
VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
2018-12-15 13:13:58.759 108055 INFO os_vif
[req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif
VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
vif_plug_ovs used the new information passed by Nova to unplug representor port:
https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809095
Title:
Wrong representor port was unplugged from OVS during cold migration
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Wrong representor port was unplugged from OVS during cold migration.
This happens when VM is scheduled to use a different PCI device to target
host vs.
to what PCI device it is using from source host. Nova uses new PCI device
information to unplug
representor port in source compute.
Steps to reproduce
==================
1. Create representor ports
$ openstack port create --network private --vnic-type=direct
--binding-profile '{"capabilities": ["switchdev"]}' direct_port1
$ openstack port create --network private --vnic-type=direct
--binding-profile '{"capabilities": ["switchdev"]}' direct_port2
2. Create VMs using the ports created above:
openstack server create --flavor m1.small --image fedora24 --nic
port-id=direct_port1 --availability-zone=nova:compute-1 vm1
openstack server create --flavor m1.small --image fedora24 --nic
port-id=direct_port2 --availability-zone=nova:compute-2 vm2
3. Migrate VM2
$ openstack server migrate vm2
$ openstack server resize --confirm vm2
4. VM2 was migrated to compute-1, however representor port is still attached
to OVS
$ sudo ovs-dpctl show
system@ovs-system:
lookups: hit:466465 missed:5411 lost:0
flows: 12
masks: hit:739146 total:2 hit/pkt:1.57
port 0: ovs-system (internal)
port 1: br-pro0.0 (internal)
port 2: br-pro0 (internal)
port 3: ens6f0
port 4: br-int (internal)
port 5: eth3
Expected result
===============
After cold migration, VM's previously used representor port should be
unplugged from OVS
Actual result
=============
VM's previously used representor port is still plugged in source host. In
some scenarios, wrong representor port was unplugged from source host. Thus
affecting VMs that were not cold migrated.
Environment
===========
Libvirt+KVM
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0
$ virsh --version
3.9.0
Neutron+OVS HW Offload
Openstack Queens openstack-nova-compute-17.0.7-1
Logs & Configs
==============
1. Plug vif device using pci address 0000:81:00.5
2018-12-15 13:12:04.871 108055 DEBUG os_vif
[req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif
VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76
2. VM was migrated from compute-1 to compute-2. New pci device is now
0000:81:00.4
2018-12-15 13:13:58.721 108055 DEBUG os_vif
[req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif
VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
2018-12-15 13:13:58.759 108055 INFO os_vif
[req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c
52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif
VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)
vif_plug_ovs used the new information passed by Nova to unplug representor
port:
https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809095/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
