Reviewed: https://review.openstack.org/527228
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=4f61f9be1704d3ed51249a10360b8c48e4cd53ff
Submitter: Zuul
Branch: master
commit 4f61f9be1704d3ed51249a10360b8c48e4cd53ff
Author: Matt Riedemann <mriedem...@gmail.com>
Date: Mon Dec 11 15:33:43 2017 -0500
Update and complete volume attachments during resize
With the new cinder volume attachment flow, during a resize the
source compute will create a new volume attachment for any BDMs
connected to the instance and delete the existing volume attachments
which represents the connection to the source host. The finish_resize
flow on the destination compute will then refresh the connection info
on the BDMs it's working with before passing them to the virt driver.
Since the source compute updated the BDM.attachment_id to point at
the new attachment which is meant for the destination host, the
refresh_connection_info call will get the connection_info for the new
reserved attachment, which is actually empty since it hasn't been
connected to the destination host yet. This results in wiping out the
BDM.connection_info which has the "driver_volume_type" value which is
something the virt driver on the destination compute needs to know
which volume backend driver to use to connect the volume to the
destination host.
This change updates the volume attachments on the destination host
before refreshing the connection_info in the BDMs and before we call
the driver to finish the resize, and then we also complete the
volume attachments once the driver has successfully finished the
resize. This is similar to what would happen via _prep_block_device
during the initial instance create, but since the code paths into the
driver are different we have to handle this explicitly in the compute
manager.
Similarly, we have to perform the same incantation when reverting a
resize and going back to the original source host.
Closes-Bug: #1737599
Change-Id: Ifc80d07d94311534fd9e7824ede9d09223a011c2
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737599
Title:
Instance resize with new-style attach volume fails
Status in OpenStack Compute (nova):
Fix Released
Bug description:
The Trove gates are failing when attempting to resize an instance what
has an ephemeral disk and an attached volume.
The stack when it fails is this:
Dec 11 03:03:28.751318 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [None req-11c69857-4556-4d83-b34c-1a0191175ceb
alt_demo alt_demo] [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Setting
instance vm_state to ERROR: VolumeDriverNotFound: Could not find a handler for
None volume.
Dec 11 03:03:28.751537 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
Traceback (most recent call last):
Dec 11 03:03:28.751683 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/compute/manager.py", line 7297, in
_error_out_instance_on_exception
Dec 11 03:03:28.751831 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
yield
Dec 11 03:03:28.751970 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/compute/manager.py", line 4358, in finish_resize
Dec 11 03:03:28.752120 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
disk_info, image_meta, bdms)
Dec 11 03:03:28.752261 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/compute/manager.py", line 4326, in _finish_resize
Dec 11 03:03:28.752408 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
old_instance_type)
Dec 11 03:03:28.752545 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220,
in __exit__
Dec 11 03:03:28.752680 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
self.force_reraise()
Dec 11 03:03:28.752813 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196,
in force_reraise
Dec 11 03:03:28.752947 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
six.reraise(self.type_, self.value, self.tb)
Dec 11 03:03:28.753092 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/compute/manager.py", line 4321, in _finish_resize
Dec 11 03:03:28.753229 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
block_device_info, power_on)
Dec 11 03:03:28.753363 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7640, in
finish_migration
Dec 11 03:03:28.753495 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
block_device_info=block_device_info)
Dec 11 03:03:28.753628 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5071, in
_get_guest_xml
Dec 11 03:03:28.753763 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
context)
Dec 11 03:03:28.753897 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4879, in
_get_guest_config
Dec 11 03:03:28.754035 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
flavor, guest.os_type)
Dec 11 03:03:28.754173 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in
_get_guest_storage_config
Dec 11 03:03:28.754303 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
self._connect_volume(connection_info, info, instance)
Dec 11 03:03:28.754429 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1180, in
_connect_volume
Dec 11 03:03:28.754597 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
vol_driver = self._get_volume_driver(connection_info)
Dec 11 03:03:28.754730 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1176, in
_get_volume_driver
Dec 11 03:03:28.754873 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
raise exception.VolumeDriverNotFound(driver_type=driver_type)
Dec 11 03:03:28.755007 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
VolumeDriverNotFound: Could not find a handler for None volume.
Dec 11 03:03:28.755150 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]:
ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]
A link to one of the gate's Nova logs is:
http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737599/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
