Public bug reported:
Description
===========
Look likes rescue may update instances.root_device_name if rescue image has
different disk bus (image property hw_disk_bus) than instance.
This introduce a mimatch between device name and driver used for instance:
During instance config generation, nova guess the disk bus driver according
table instance_system_metadata.image_hw_disk_bus,
and get root device name from table instances.root_device_name.
Because of this mismatch, cinder attachment may failed with the following error
message in compute log:
unable to execute QEMU command 'device_add': Duplicate ID 'virtio-disk0' for
device
Probable solution is to avoid rescue action to update
instance.root_device_name
Steps to reproduce
==================
On a fresh master devstack:
openstack image save cirros-0.4.0-x86_64-disk --file
/tmp/cirros-0.4.0-x86_64-disk.disk
#create a new image, but an scsi one:
openstack image create --container-format bare --disk-format qcow2 --file
/tmp/cirros-0.4.0-x86_64-disk.disk --property hw_disk_bus='scsi' --property
hw_scsi_model='virtio-scsi' cirros-0.4.0-x86_64-scsi-disk
#create instance with default virtio driver:
openstack server create --flavor m1.small --image cirros-0.4.0-x86_64-disk
--nic net-id=private test
mysql> select root_device_name from instances where uuid='xxx'
/dev/vda
#rescue instance but with the scsi image:
$openstack server rescue xxxx --image cirros-0.4.0-x86_64-scsi-disk
mysql> select root_device_name from instances where uuid='xxx'
/dev/sda
$openstack server unrescue xxxx
# root_device_name is still on sda should be on vda according instance metadata
mysql> select root_device_name from instances where uuid='xxx'
/dev/sda
$virsh dumpxml instance-00000001 | grep "bus='virtio"
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
# at the next hard reboot new xml is generated with scsi device name BUT with
virtio driver.
$openstack server reboot --hard xxx
$virsh dumpxml instance-00000001 | grep -A 1 "bus='virtio"
<target dev='sda' bus='virtio'/>
<alias name='virtio-disk0'/>
$openstack volume create --size 10 test
$openstack server add volume 1c9b1582-5fc7-417a-a8a0-387e8833731f
0621430c-b0d2-4cca-8868-f86f36f1ef29
$sudo journalctl -u devstack@n-cpu.service | grep Duplicate
Jul 05 09:29:54 alex-devstack-compute2 nova-compute[28285]: ERROR
nova.virt.libvirt.driver [None req-38714989-4deb-4a05-bdfc-3418edbda7e3 demo
demo] [instance: 1c9b1582-5fc7-417a-a8a0-387e8833731f] Failed to attach volume
at mountpoint: /dev/vda: libvirtError: internal error: unable to execute QEMU
command 'device_add': Duplicate ID 'virtio-disk0' for device
Error probably comes from the fact that nova lookup for next availiable virtio
device based on name, which is vda - virtio-disk0 (as root device is currently
sda)
but because root device sda is already using virtio-disk0 it failed.
Expected result
===============
instance root_device_name should remain the same as before rescue/unrescue,
regardless of image used for rescuing.
Actual result
=============
instance root_device_name is updated according the hw_disk_bus property for the
image used during rescue(and never set back to original value)
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1835926
Title:
Volume attachment may fail after rescuing instance on an image with
different hw_disk_bus
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Look likes rescue may update instances.root_device_name if rescue image has
different disk bus (image property hw_disk_bus) than instance.
This introduce a mimatch between device name and driver used for instance:
During instance config generation, nova guess the disk bus driver according
table instance_system_metadata.image_hw_disk_bus,
and get root device name from table instances.root_device_name.
Because of this mismatch, cinder attachment may failed with the following
error message in compute log:
unable to execute QEMU command 'device_add': Duplicate ID 'virtio-disk0' for
device
Probable solution is to avoid rescue action to update
instance.root_device_name
Steps to reproduce
==================
On a fresh master devstack:
openstack image save cirros-0.4.0-x86_64-disk --file
/tmp/cirros-0.4.0-x86_64-disk.disk
#create a new image, but an scsi one:
openstack image create --container-format bare --disk-format qcow2 --file
/tmp/cirros-0.4.0-x86_64-disk.disk --property hw_disk_bus='scsi' --property
hw_scsi_model='virtio-scsi' cirros-0.4.0-x86_64-scsi-disk
#create instance with default virtio driver:
openstack server create --flavor m1.small --image cirros-0.4.0-x86_64-disk
--nic net-id=private test
mysql> select root_device_name from instances where uuid='xxx'
/dev/vda
#rescue instance but with the scsi image:
$openstack server rescue xxxx --image cirros-0.4.0-x86_64-scsi-disk
mysql> select root_device_name from instances where uuid='xxx'
/dev/sda
$openstack server unrescue xxxx
# root_device_name is still on sda should be on vda according instance
metadata
mysql> select root_device_name from instances where uuid='xxx'
/dev/sda
$virsh dumpxml instance-00000001 | grep "bus='virtio"
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
# at the next hard reboot new xml is generated with scsi device name BUT with
virtio driver.
$openstack server reboot --hard xxx
$virsh dumpxml instance-00000001 | grep -A 1 "bus='virtio"
<target dev='sda' bus='virtio'/>
<alias name='virtio-disk0'/>
$openstack volume create --size 10 test
$openstack server add volume 1c9b1582-5fc7-417a-a8a0-387e8833731f
0621430c-b0d2-4cca-8868-f86f36f1ef29
$sudo journalctl -u devstack@n-cpu.service | grep Duplicate
Jul 05 09:29:54 alex-devstack-compute2 nova-compute[28285]: ERROR
nova.virt.libvirt.driver [None req-38714989-4deb-4a05-bdfc-3418edbda7e3 demo
demo] [instance: 1c9b1582-5fc7-417a-a8a0-387e8833731f] Failed to attach volume
at mountpoint: /dev/vda: libvirtError: internal error: unable to execute QEMU
command 'device_add': Duplicate ID 'virtio-disk0' for device
Error probably comes from the fact that nova lookup for next availiable
virtio device based on name, which is vda - virtio-disk0 (as root device is
currently sda)
but because root device sda is already using virtio-disk0 it failed.
Expected result
===============
instance root_device_name should remain the same as before rescue/unrescue,
regardless of image used for rescuing.
Actual result
=============
instance root_device_name is updated according the hw_disk_bus property for
the image used during rescue(and never set back to original value)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1835926/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
