Public bug reported: Resize revertion leaves instance directory on the second host with Ceph image backend. As the result the second attempt to resize the instance to the same host fails with n-cpu.log:
Traceback (most recent call last): File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize disk_info, image_meta) File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize old_instance_type) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize block_device_info, power_on) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration self._ensure_console_log_for_instance(instance) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance libvirt_utils.file_open(console_file, 'a').close() File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open return open(*args, **kwargs) IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log' $ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/ total 24 -rw-rw-r--. 1 qemu qemu 19342 Oct 7 21:23 console.log -rw-rw-r--. 1 stack libvirtd 2762 Oct 7 21:22 libvirt.xml Steps to reproduce: 1 Run 2-nodes devstack with Ceph image backend 2 Run an instance $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1 3 Disable the instance host $ nova service-disable 172.16.1.10 nova-compute 4 Resize the instance to another host $ nova migrate inst-1 5 Revert resize $ nova resize-revert inst-1 6 Resize the instance again $ nova migrate inst-1 7 Check the instance state Actual result - the instance is in error state. Expected result - the instance is in verify_resize state. Check n-cpu.log on the second node, where the instance was migrated. This has been reproduced on master commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015 Merge: 870a77f 453e71d Author: Jenkins <jenk...@review.openstack.org> Date: Wed Oct 5 01:35:48 2016 +0000 ** Affects: nova Importance: Undecided Status: New ** Description changed: Resize revertion leaves instance directory on the second host with Ceph image backend. As the result the second attempt to resize the instance to the same host fails with n-cpu.log: Traceback (most recent call last): - File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize - disk_info, image_meta) - File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize - old_instance_type) - File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ - self.force_reraise() - File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise - six.reraise(self.type_, self.value, self.tb) - File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize - block_device_info, power_on) - File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration - self._ensure_console_log_for_instance(instance) - File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance - libvirt_utils.file_open(console_file, 'a').close() - File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open - return open(*args, **kwargs) + File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize + disk_info, image_meta) + File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize + old_instance_type) + File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ + self.force_reraise() + File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise + six.reraise(self.type_, self.value, self.tb) + File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize + block_device_info, power_on) + File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration + self._ensure_console_log_for_instance(instance) + File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance + libvirt_utils.file_open(console_file, 'a').close() + File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open + return open(*args, **kwargs) IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log' $ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/ total 24 -rw-rw-r--. 1 qemu qemu 19342 Oct 7 21:23 console.log -rw-rw-r--. 1 stack libvirtd 2762 Oct 7 21:22 libvirt.xml Steps to reproduce: 1 Run 2-nodes devstack with Ceph image backend 2 Run an instance - $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1 + $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1 3 Disable the instance host - $ nova service-disable 172.16.1.10 nova-compute + $ nova service-disable 172.16.1.10 nova-compute 4 Resize the instance to another host - $ nova migrate inst-1 + $ nova migrate inst-1 5 Revert resize - $ nova resize-revert inst-1 + $ nova resize-revert inst-1 6 Resize the instance again - $ nova migrate inst-1 + $ nova migrate inst-1 7 Check the instance state Actual result - the instance is in error state. Expected result - the instance is in verify_resize state. Check n-cpu.log on the second node, where the instance was migrated. + + This has been reproduced on master + commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015 + Merge: 870a77f 453e71d + Author: Jenkins <jenk...@review.openstack.org> + Date: Wed Oct 5 01:35:48 2016 +0000 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1631481 Title: Revert resize does not delete instance directory with Ceph Status in OpenStack Compute (nova): New Bug description: Resize revertion leaves instance directory on the second host with Ceph image backend. As the result the second attempt to resize the instance to the same host fails with n-cpu.log: Traceback (most recent call last): File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize disk_info, image_meta) File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize old_instance_type) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize block_device_info, power_on) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration self._ensure_console_log_for_instance(instance) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance libvirt_utils.file_open(console_file, 'a').close() File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open return open(*args, **kwargs) IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log' $ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/ total 24 -rw-rw-r--. 1 qemu qemu 19342 Oct 7 21:23 console.log -rw-rw-r--. 1 stack libvirtd 2762 Oct 7 21:22 libvirt.xml Steps to reproduce: 1 Run 2-nodes devstack with Ceph image backend 2 Run an instance $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1 3 Disable the instance host $ nova service-disable 172.16.1.10 nova-compute 4 Resize the instance to another host $ nova migrate inst-1 5 Revert resize $ nova resize-revert inst-1 6 Resize the instance again $ nova migrate inst-1 7 Check the instance state Actual result - the instance is in error state. Expected result - the instance is in verify_resize state. Check n-cpu.log on the second node, where the instance was migrated. This has been reproduced on master commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015 Merge: 870a77f 453e71d Author: Jenkins <jenk...@review.openstack.org> Date: Wed Oct 5 01:35:48 2016 +0000 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1631481/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp