** Also affects: nova/pike
Importance: Undecided
Status: New
** Changed in: nova/pike
Status: New => In Progress
** Changed in: nova/pike
Importance: Undecided => Low
** Changed in: nova/pike
Assignee: (unassigned) => Wangpan (aspirer)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734838
Title:
rotate backups failed because of image in use
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) pike series:
In Progress
Bug description:
This issue similar to https://bugs.launchpad.net/nova/+bug/1634773,
Reproduce steps:
1. create an instance with system disk in rbd backend, named
Instance-for-backup
2. create the first backup of this instance, such as: nova backup
Instance-for-backup backup-1 daily 2
3. create a new instance with this backup-1 snapshot image, such as: nova
boot --flavor m1.tiny --image backup-1 ...
4. then create the second backup of Instance-for-backup, CLI is similar as
step 2
5. finally we create the third backup, we expect the backup-1 image should be
rotated out by nova, but it doesn't, and nova-compute report an exception in
it's log:
2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance:
00bce146-5408-4c33-bc11-7458f847eb19] Rotating out 52 backups _rotate_back
ups /usr/lib/python2.7/site-packages/nova/compute/manager.py:3262
2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance:
00bce146-5408-4c33-bc11-7458f847eb19] Deleting image 9a993e71-71ca-490e-b3
cb-0b4dca2e574c _rotate_backups
/usr/lib/python2.7/site-packages/nova/compute/manager.py:3267
2017-11-28 16:29:19.280 4154 DEBUG oslo_concurrency.lockutils
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources"
acquired by "nova.compute.resource_tracker.update_usag
e" :: waited 0.000s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-28 16:29:19.353 4154 DEBUG oslo_concurrency.lockutils
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources"
released by "nova.compute.resource_tracker.update_usag
e" :: held 0.074s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-28 16:29:19.354 4154 INFO nova.compute.manager
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad7
c 0e5bcf89990c455882649ed88b32e27d - - -] [instance:
00bce146-5408-4c33-bc11-7458f847eb19] Successfully reverted task state from
None on failure for instance.
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
[req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e
742bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Exception during message
handling: 409 Conflict: Image 9a993e71-71ca-490e-b3
cb-0b4dca2e574c could not be deleted because it is in use: The image cannot
be deleted because it is in use through the backend
store outside of Glance. (HTTP 409)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
patcher.py", line 138, in _dispatch_and_reply
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
incoming.message))
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
patcher.py", line 185, in _dispatch
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
patcher.py", line 127, in _do_dispatch
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher result =
func(ctxt, **new_args)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", li
ne 110, in wrapped
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher payload)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
", line 220, in __exit__
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
", line 196, in force_reraise
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
LOG.warning(msg, e, instance=instance)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3124, in
backup_instance
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self._rotate_backups(context, instance, backup_type, rotation)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in
decorated_function
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3269, in
_rotate_backups
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self.image_api.delete(context, image_id)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/api.py", line 141, in delete
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
session.delete(context, image_id)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/glance.py", line 499, in delete
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
self._client.call(context, 1, 'delete', image_id)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/glance.py", line 250, in call
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher result =
getattr(client.images, method)(*args, **kwargs)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 291, in
delete
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher resp,
body = self.client.delete(url)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 287, in
delete
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher return
self._request('DELETE', url, **kwargs)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 267, in
_request
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher resp,
body_iter = self._handle_response(resp)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 83, in
_handle_response
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher raise
exc.from_response(resp, resp.content)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
HTTPConflict: 409 Conflict: Image 9a993e71-71ca-490e-b3cb-0b4dca2e574c could
not be deleted because it is in use: The image cannot be deleted because it is
in use through the backend store outside of Glance. (HTTP 409)
2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1734838/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
