This is likely a side effect of the fix for bug 1728603
https://review.openstack.org/#/c/516395/.
** Tags added: libvirt resize
** Changed in: nova
Status: New => Triaged
** Summary changed:
- After cold-migration, disk.info file leftover on source host
+ After cold-migration of a volume-backed instance, disk.info file leftover on
source host
** Also affects: nova/ocata
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
** Also affects: nova/pike
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1769131
Title:
After cold-migration of a volume-backed instance, disk.info file
leftover on source host
Status in OpenStack Compute (nova):
Triaged
Status in OpenStack Compute (nova) ocata series:
New
Status in OpenStack Compute (nova) pike series:
New
Status in OpenStack Compute (nova) queens series:
New
Bug description:
Tested using kolla-ansible, with kolla images stable/queens.
In this setup there are only two compute nodes, with cinder/lvm for
storage.
A cirros instance is created, on compute02, and cold-migrated to
compute01.
At the step where it's awaiting confirmation, the following files can
be found:
compute01
/var/lib/docker/volumes/nova_compute/_data/instances
\-- 371e669b-0f15-49f2-9a84-bd1e89f34294
\-- console.log
compute02
1 directory, 1 file
/var/lib/docker/volumes/nova_compute/_data/instances
\-- 371e669b-0f15-49f2-9a84-bd1e89f34294_resize
\-- console.log
1 directory, 1 file
After confirming the migrate/resize, this becomes:
compute01
/var/lib/docker/volumes/nova_compute/_data/instances
\-- 371e669b-0f15-49f2-9a84-bd1e89f34294
\-- console.log
compute02
1 directory, 1 file
/var/lib/docker/volumes/nova_compute/_data/instances
\-- 371e669b-0f15-49f2-9a84-bd1e89f34294
\-- disk.info
1 directory, 1 file
This log shows how after the _resize information is cleaned up, that
*after this, this file ends up on the source host, where it is left.
http://paste.openstack.org/show/720358/
2018-05-04 12:55:10.818 7 DEBUG nova.compute.manager
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] Going to confirm migration 4
do_confirm_resize /usr/lib/python2.7/site-packages/nova/compute/manager.py:3684
2018-05-04 12:55:11.032 7 DEBUG oslo_concurrency.lockutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Acquired semaphore
"refresh_cache-371e669b-0f15-49f2-9a84-bd1e89f34294" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2018-05-04 12:55:11.033 7 DEBUG nova.network.neutronv2.api
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] _get_instance_nw_info()
_get_instance_nw_info
/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1383
2018-05-04 12:55:11.034 7 DEBUG nova.objects.instance
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Lazy-loading 'info_cache'
on Instance uuid 371e669b-0f15-49f2-9a84-bd1e89f34294 obj_load_attr
/usr/lib/python2.7/site-packages/nova/objects/instance.py:1052
2018-05-04 12:55:11.406 7 DEBUG nova.network.base_api
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] Updating instance_info_cache with
network_info: [{"profile": {}, "ovs_interfaceid":
"ba8646b4-fa66-46b9-9f7e-a83163668bb8", "preserve_on_delete": false, "network":
{"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type":
"fixed", "floating_ips": [], "address": "10.0.0.8"}], "version": 4, "meta":
{"dhcp_server": "10.0.0.2"}, "dns": [{"meta": {}, "version": 4, "type": "dns",
"address": "8.8.8.8"}], "routes": [], "cidr": "10.0.0.0/24", "gateway":
{"meta": {}, "version": 4, "type": "gateway", "address": "10.0.0.1"}}], "meta":
{"injected": false, "tenant_id": "7ea70c4f74c24199b14df0a570b6f93e", "mtu":
1450}, "id": "f1d14432-5a26-4b0a-89e7-6683bd7d2477", "label": "demo-net"},
"devname": "tapba8646b4-fa", "vnic_type": "normal", "qbh_params": null, "meta":
{}, "details": {"port_filter": true, "datapath_type": "system",
"ovs_hybrid_plug": true}, "address": "fa:16:3e:d9:91:37", "active": true,
"type": "ovs", "id": "ba8646b4-fa66-46b9-9f7e-a83163668bb8", "qbg_params":
null}] update_instance_cache_with_nw_info
/usr/lib/python2.7/site-packages/nova/network/base_api.py:48
2018-05-04 12:55:11.426 7 DEBUG oslo_concurrency.lockutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Releasing semaphore
"refresh_cache-371e669b-0f15-49f2-9a84-bd1e89f34294" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:228
2018-05-04 12:55:11.426 7 DEBUG oslo_concurrency.processutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Running cmd (subprocess):
rm -rf /var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294_resize
execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
2018-05-04 12:55:11.459 7 DEBUG oslo_concurrency.processutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] CMD "rm -rf
/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294_resize" returned:
0 in 0.033s execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409
2018-05-04 12:55:11.462 7 DEBUG oslo_concurrency.lockutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Lock
"/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info"
acquired by "nova.virt.libvirt.imagebackend.write_to_disk_info_file" :: waited
0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2018-05-04 12:55:11.462 7 DEBUG oslo_concurrency.lockutils
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] Lock
"/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info"
released by "nova.virt.libvirt.imagebackend.write_to_disk_info_file" :: held
0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2018-05-04 12:55:11.482 7 DEBUG nova.virt.libvirt.vif
[req-510561e2-eabb-4c37-8fc3-d56e9f50bf6e 64ca3042227c48ea84d77461b14b8acb
7ea70c4f74c24199b14df0a570b6f93e - default default] vif_type=ovs
instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2018-05-04T11:53:34Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description=None,display_name='cirros',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(2),host='compute01',hostname='cirros',id=2,image_ref='',info_cache=InstanceInfoCache,instance_type_id=2,kernel_id='',key_data='ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQDGUK82VwkyVJoNMlF5EhqfVaI+yOfhaMnMWbLg6ZDeKQjJ5gTZ7DvAfF2NOsyY9kYVo2ik3tQiVJmTyQbc4zQZN327PgnHm4HkmQUTx/pz57VfXzpGg1lQviGW8wr7+Pd7euMcazt2eZB3l4dL1xL96dSIoBzK0wG7B4KTEk8uWMhFkhVFrH6LQBtJSkrTkPWIafc3fv3XNhs4bo9mXQNOpWW6pJogx6FiPYqkFtynHdJTX0a/JcdJxmu/HPSwT3QmZ3yyasHQ1+It6Htte0P1ThdsMKavRD9Gki/r5cB2sUxUxbfSFMfiHdry7opefrbvRVU3G1xwKqrd9JdCCDe9
kolla@operator
This file should not be left on the source host.
For example, attempting to live-migrate back to this host results in a
failure:
2018-05-04 13:45:40.546 7 ERROR nova.compute.manager [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7407, in
pre_live_migration
2018-05-04 13:45:40.546 7 ERROR nova.compute.manager [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] raise
exception.DestinationDiskExists(path=instance_dir)
2018-05-04 13:45:40.546 7 ERROR nova.compute.manager [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294]
2018-05-04 13:45:40.546 7 ERROR nova.compute.manager [instance:
371e669b-0f15-49f2-9a84-bd1e89f34294] DestinationDiskExists: The supplied disk
path (/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294) already
exists, it is expected not to exist.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1769131/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
