Reviewed: https://review.openstack.org/388519 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cc5d3437cabab4688df8492e071ea8c597311113 Submitter: Jenkins Branch: master
commit cc5d3437cabab4688df8492e071ea8c597311113 Author: bhagyashris <[email protected]> Date: Mon Oct 17 19:29:21 2016 +0530 Handle ImageNotFound exception during instance backup If user have already backed up instance to few numbers and then execute backup api with rotation 1, then nova will delete the previously created images exceeding rotation limit. During deleting these images, if user mistakenly deletes one of the image in advance, then backup instance won't be able to delete all images exceeding rotation limit causing api failure. This patch handles ImageNotFound exception during deleting backup images, logs a warning message and continues deleting all of the remaining images. Closes-Bug: #1634773 Change-Id: Ib999138ddbbcd62e0c99a69e5c6d37b4e49f871f ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1634773 Title: backup instance won't be able to delete all images exceeding rotation limit, if user mistakenly deletes one of the backup image Status in OpenStack Compute (nova): Fix Released Bug description: If user have already backed up instance to few numbers and then execute backup api with rotation 1, then nova will delete the previously created images exceeding rotation limit. During deleting these images, if user mistakenly deletes one of the image in advance, then backup instance won't be able to delete all images exceeding rotation limit causing api failure. Steps to reproduce: 1. Create instance $ nova boot --flavor <flavor_id> --image <image_id> <server_name> $ nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec test +--------------------------------------+------+--------+------------+-------------+---------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+---------------------------------+ | a9e41bc5-516c-48f1-bed8-2475650835c8 | test | ACTIVE | - | Running | public=2001:db8::8, 172.24.4.12 | +--------------------------------------+------+--------+------------+-------------+---------------------------------+ 2. Create 5 instance backup using the backup api(hit below command 5 times) $ nova backup <server> <name> <backup-type> <rotation> $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 5 3. Take list of backup images $ glance image-list +--------------------------------------+---------------------------------+ | ID | Name | +--------------------------------------+---------------------------------+ | c33790be-df8b-4740-8311-0fa787bfb087 | cirros-0.3.4-x86_64-uec | | de43a375-6d6d-4a30-856e-7a68db083834 | cirros-0.3.4-x86_64-uec-kernel | | 0a6c800e-6002-4f2f-a2b9-0542e0f2fac0 | cirros-0.3.4-x86_64-uec-ramdisk | | fc0d2030-1cec-4780-9199-e99c63da0005 | test_bk | | a9d7dc78-acab-4e4e-b03d-c7da9cb5ef00 | test_bk | | 937061b4-51c7-4835-b9b7-d63464665a27 | test_bk | | 41e25c26-c61e-421c-b170-a39539bcbc78 | test_bk | | a15ae46a-a3c3-49a8-be75-1aee724fa4e9 | test_bk | +--------------------------------------+---------------------------------+ 4. Add debug point using pdb.set_trace() in nova.compute.manager._rotate_backups method [1] 5. Delete one of the backup images from above list $ glance image-delete fc0d2030-1cec-4780-9199-e99c63da0005 6. Create the insatnce backup by passing the rotation parameter value as 1. $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 1 7. Checked the n-cpu logs 2016-10-17 15:59:38.978 ERROR oslo_messaging.rpc.server [req-99fcf837-514a-4a80-a19e-c8ac2f26768f admin admin] Exception during message handling 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server Traceback (most recent call last): 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server function_name, call_dict, binary) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise() 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 179, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server instance=instance) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise() 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise() 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3001, in backup_instance 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self._rotate_backups(context, instance, backup_type, rotation) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise() 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3145, in _rotate_backups 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.image_api.delete(context, image_id) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/api.py", line 141, in delete 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return session.delete(context, image_id) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 765, in delete 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server raise exception.ImageNotFound(image_id=image_id) 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server ImageNotFound: Image fc0d2030-1cec-4780-9199-e99c63da0005 could not be found. 2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server [1] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3139 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1634773/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
