Public bug reported:
We have flavor classes using different nova disk backends that are
separated using host aggregates. For example, we have a flavor named
l1.tiny which is using imagebackend, and s1.small using rbd backend. The
hypervisors configured for imagebackend are added to the host aggregate
where l1.* instances are scheduled, and the rbd hypervisors are in an
aggregate where s1.* instances are scheduled.
When resizing an instance from l1.tiny to s1.small, the instance fails
to resize and enters error state. The root disk is also lost during the
failed resize. The host of the instance is set to one of the s1.*
aggregate HVs, and the imagebackend disk is no longer present on the
original l1.* hypervisor.
The error provided in 'instance show' is:
| fault | {u'message': u'[errno 2] error
opening image 5a8ab7a3-3e59-442c-a603-2c24652788cb_disk at snapshot
None', u'code': 500, u'details': u' File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 204, in decorated_function\n return function(self, context,
*args, **kwargs)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 4062, in finish_resize\n
self._set_instance_obj_error_state(context, instance)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/compute/manager.py", line 4050, in finish_resize\n
disk_info, image_meta)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 4012, in _finish_resize\n old_instance_type)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/compute/manager.py", line 4007, in _finish_resize\n
block_device_info, power_on)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
line 7454, in finish_migration\n
fallback_from_host=migration.source_compute)\n File "/openstack/venvs
/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/driver.py", line 3160, in _create_image\n
fallback_from_host)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
line 3264, in _create_and_inject_local_root\n
backend.create_snap(libvirt_utils.RESIZE_SNAPSHOT_NAME)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/imagebackend.py", line 941, in create_snap\n
return self.driver.create_snap(self.rbd_name, name)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/storage/rbd_utils.py", line 392, in
create_snap\n with RBDVolumeProxy(self, str(volume), pool=pool) as
vol:\n File "/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/storage/rbd_utils.py", line 78, in __init__\n
driver._disconnect_from_rados(client, ioctx)\n File "/openstack/venvs
/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 74, in
__init__\n read_only=read_only))\n File "rbd.pyx", line 1392, in
rbd.Image.__init__ (/build/ceph-12.2.2/obj-x86_64-linux-
gnu/src/pybind/rbd/pyrex/rbd.c:13540)\n', u'created':
u'2018-11-14T11:03:11Z'} |
We are currently seeing this behavior on Ocata. I'm not certain if more recent
nova releases experience this also.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1803331
Title:
Root disk lost when resizing instance from imagebackend to rbd backed
flavor
Status in OpenStack Compute (nova):
New
Bug description:
We have flavor classes using different nova disk backends that are
separated using host aggregates. For example, we have a flavor named
l1.tiny which is using imagebackend, and s1.small using rbd backend.
The hypervisors configured for imagebackend are added to the host
aggregate where l1.* instances are scheduled, and the rbd hypervisors
are in an aggregate where s1.* instances are scheduled.
When resizing an instance from l1.tiny to s1.small, the instance fails
to resize and enters error state. The root disk is also lost during
the failed resize. The host of the instance is set to one of the s1.*
aggregate HVs, and the imagebackend disk is no longer present on the
original l1.* hypervisor.
The error provided in 'instance show' is:
| fault | {u'message': u'[errno 2] error
opening image 5a8ab7a3-3e59-442c-a603-2c24652788cb_disk at snapshot
None', u'code': 500, u'details': u' File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 204, in decorated_function\n return function(self, context,
*args, **kwargs)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 4062, in finish_resize\n
self._set_instance_obj_error_state(context, instance)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/compute/manager.py", line 4050, in finish_resize\n
disk_info, image_meta)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/nova/compute/manager.py",
line 4012, in _finish_resize\n old_instance_type)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/compute/manager.py", line 4007, in
_finish_resize\n block_device_info, power_on)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/driver.py", line 7454, in
finish_migration\n fallback_from_host=migration.source_compute)\n
File "/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/driver.py", line 3160, in _create_image\n
fallback_from_host)\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/driver.py", line 3264, in
_create_and_inject_local_root\n
backend.create_snap(libvirt_utils.RESIZE_SNAPSHOT_NAME)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/imagebackend.py", line 941, in
create_snap\n return self.driver.create_snap(self.rbd_name, name)\n
File "/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/storage/rbd_utils.py", line 392, in
create_snap\n with RBDVolumeProxy(self, str(volume), pool=pool) as
vol:\n File "/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/nova/virt/libvirt/storage/rbd_utils.py", line 78, in
__init__\n driver._disconnect_from_rados(client, ioctx)\n File
"/openstack/venvs/nova-untagged/local/lib/python2.7/site-
packages/oslo_utils/excutils.py", line 220, in __exit__\n
self.force_reraise()\n File "/openstack/venvs/nova-
untagged/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise\n six.reraise(self.type_, self.value,
self.tb)\n File "/openstack/venvs/nova-untagged/local/lib/python2.7
/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 74, in
__init__\n read_only=read_only))\n File "rbd.pyx", line 1392, in
rbd.Image.__init__ (/build/ceph-12.2.2/obj-x86_64-linux-
gnu/src/pybind/rbd/pyrex/rbd.c:13540)\n', u'created':
u'2018-11-14T11:03:11Z'} |
We are currently seeing this behavior on Ocata. I'm not certain if more
recent nova releases experience this also.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1803331/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
