Public bug reported:
I'm using OpenStack Victoria.
When I try to take snapshots of instances above a certain size, they
disappear during saving state and glance shows the following error:
Mar 30 09:15:14 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:15:14.306 96 INFO glance.api.v2.image_data
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 30 09:16:10 infra1-glance-container-99614ac2 uwsgi[96]: Tue Mar 30 09:16:10
2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client
disconnected) on request /v2/images/c83f6c4d-0083-4600-95f5-d21b4623af47/file
(ip 192.168.110.214) !!!
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.427 96 ERROR glance_store._drivers.swift.store
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload
to backend, deleting stale chunks.: swiftclient.exceptions.ClientException:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.494 96 ERROR glance_store._drivers.swift.store
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to
Swift.
Got error
from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..: swiftclient.exceptions.ClientException:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.497 96 ERROR glance.api.v2.image_data
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data
due to internal error: glance_store.exceptions.BackendException: Failed to add
object to Swift.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add
object to Swift.
Got error
from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..: glance_store.exceptions.BackendException: Failed
to add object to Swift.
Got error
from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call
last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1014, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._delete_stale_chunks(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1003, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
manager.get_connection().put_object(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1960, in put_object
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
self._retry(reset_func, put_object, container, obj, contents,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1843, in _retry
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi reset_func(func, *args,
**kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1940, in _default_reset
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise
ClientException('put_object(%r, %r, ...) failure and no '
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
swiftclient.exceptions.ClientException: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi During handling of the
above exception, another exception occurred:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call
last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1347, in __call__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi action_result =
self.dispatch(self.controller, action,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1391, in dispatch
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return method(*args,
**kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py",
line 416, in wrapped
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return func(self, req,
*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 298, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._restore(image_repo, image)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 163, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi image.set_data(data,
size, backend=backend)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py",
line 208, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.base.set_data(data, size, backend=backend, set_active=set_active)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 501, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
_send_notification(notify_error, 'image.upload', msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 447, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.repo.set_data(data, size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py",
line 198, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
self.image.set_data(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py",
line 318, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.image.set_data(data, size=size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 567, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._upload_to_store(data, verifier, backend, size)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 458, in _upload_to_store
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi multihash, loc_meta) =
self.store_api.add_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 398, in add_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
store_add_to_backend_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 480, in store_add_to_backend_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi (location, size,
checksum, multihash, metadata) = store.add(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py",
line 279, in add_adapter
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi metadata_dict) =
store_add_fun(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py",
line 176, in op_checker
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
store_op_fun(store, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1082, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise
glance_store.BackendException(msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
glance_store.exceptions.BackendException: Failed to add object to Swift.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Got error from Swift:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
I don't see errors in the swift logs:
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift
- - - - txf319964369254af4974a7-006062ec22 - 0.0135 RL - 1617095714.876815081
1617095714.890305758 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift
- - - - txf319964369254af4974a7-006062ec22 - 0.0131 CS - 1617095714.892540216
1617095714.905595303 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting
HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0127 CS -
1617095714.907815695 1617095714.920493126 0
Mar 30 09:15:15 infra1-swift-proxy-container-27169fa7 proxy-server[1068]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/15/15 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting
HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - -
txf319964369254af4974a7-006062ec22 - 0.2779 - - 1617095714.874005318
1617095715.151889324 0
Mar 30 09:15:17 infra1-swift-proxy-container-27169fa7 proxy-server[1071]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/15/17 PUT
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001
HTTP/1.0 201 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 - -
txc05fff07e3384a68a7fa9-006062ec23 - 2.8218 - - 1617095715.163244247
1617095717.985031843 0
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]:
Client disconnected without sending last chunk (txn:
tx49b0efe394364bf1975bb-006062ec25) (client_ip: 192.168.110.106)
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/16/10 PUT
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002
HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 89 -
tx49b0efe394364bf1975bb-006062ec25 - 52.2224 - - 1617095717.995234489
1617095770.217619896 0
Mar 30 09:16:11 infra1-swift-proxy-container-27169fa7 proxy-server[1080]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/16/11 DELETE
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001
HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - -
tx98a9247419a2445fafcbf-006062ec5b - 0.0486 - - 1617095771.441193104
1617095771.489775896 0
and
Mar 30 09:15:17 bc1bl12 object-server[694851]: 10.0.3.211 - -
[30/Mar/2021:09:15:17 +0000] "PUT
/os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"
201 - "PUT
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001";
"txc05fff07e3384a68a7fa9-006062ec23" "proxy-server 1071" 2.8030 "-" 694851 0
Mar 30 09:16:10 bc1bl12 object-server[694850]: 10.0.3.211 - -
[30/Mar/2021:09:16:10 +0000] "PUT
/os-objects/43/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002"
499 89 "PUT
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002";
"tx49b0efe394364bf1975bb-006062ec25" "proxy-server 1067" 52.1551 "-" 694850 0
Mar 30 09:16:11 bc1bl12 object-server[694854]: 10.0.3.211 - -
[30/Mar/2021:09:16:11 +0000] "DELETE
/os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"
204 - "DELETE
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001";
"tx98a9247419a2445fafcbf-006062ec5b" "proxy-server 1080" 0.0321 "-" 694854 0
nova-compute shows the following:
Mar 30 09:13:24 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:24.975 650019
INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] instance snapshotting
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.038 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Beginning live
snapshot process
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.527 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Skipping quiescing
instance: QEMU guest agent is not enabled.
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.384 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.386 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.983 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.390 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.391 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:14:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:58.006 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:15:13 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:13.861 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Snapshot extracted,
beginning image upload
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.395 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.397 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.997 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.530 650019
INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted
task state from image_uploading on failure for instance.
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Exception during message handling:
glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The
server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2478, in snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server metadata['location'] =
root_disk.direct_snapshot(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py",
line 452, in direct_snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise
NotImplementedError(_('direct_snapshot() is not implemented'))
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not
implemented
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server During handling of the above exception, another
exception occurred:
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method,
ctxt, args)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 76, in wrapped
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 69, in wrapped
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 188, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state
for instance. "
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 159, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py",
line 1434, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 216, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 205, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 236, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server compute_utils.delete_image(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 232, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, image_id,
instance,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3908, in snapshot_instance
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id,
instance,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3941, in _snapshot_instance
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance,
image_id,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2549, in snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self._image_api.update(context,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1247, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return session.update(context, image_id,
image_info, data=data,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 696, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1035, in _reraise_translated_image_exception
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 694, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server image = self._update_v2(context,
sent_service_image_meta, data)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 713, in _update_v2
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server image = self._upload_data(context,
image_id, data)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 589, in _upload_data
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line
694, in tpool_execute
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 192, in call
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py",
line 600, in inner
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args,
**kwargs))
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py",
line 289, in upload
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url,
headers=hdrs, data=body)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py",
line 404, in put
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 380, in request
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self._handle_response(resp)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 120, in _handle_response
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation.
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.391 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.392 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.997 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
** Affects: glance
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1921893
Title:
Snapshots fail: Failed to add object to Swift
Status in Glance:
New
Bug description:
I'm using OpenStack Victoria.
When I try to take snapshots of instances above a certain size, they
disappear during saving state and glance shows the following error:
Mar 30 09:15:14 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:15:14.306 96 INFO glance.api.v2.image_data
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 30 09:16:10 infra1-glance-container-99614ac2 uwsgi[96]: Tue Mar 30
09:16:10 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the
client disconnected) on request
/v2/images/c83f6c4d-0083-4600-95f5-d21b4623af47/file (ip 192.168.110.214) !!!
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.427 96 ERROR glance_store._drivers.swift.store
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload
to backend, deleting stale chunks.: swiftclient.exceptions.ClientException:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.494 96 ERROR glance_store._drivers.swift.store
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to
Swift.
Got
error from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..: swiftclient.exceptions.ClientException:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.497 96 ERROR glance.api.v2.image_data
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data
due to internal error: glance_store.exceptions.BackendException: Failed to add
object to Swift.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
[req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add
object to Swift.
Got
error from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..: glance_store.exceptions.BackendException: Failed
to add object to Swift.
Got
error from Swift: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call
last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1014, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._delete_stale_chunks(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1003, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
manager.get_connection().put_object(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1960, in put_object
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
self._retry(reset_func, put_object, container, obj, contents,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1843, in _retry
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi reset_func(func, *args,
**kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1940, in _default_reset
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise
ClientException('put_object(%r, %r, ...) failure and no '
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
swiftclient.exceptions.ClientException: put_object('glance_images',
'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to
reset contents for reupload.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi During handling of the
above exception, another exception occurred:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call
last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1347, in __call__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi action_result =
self.dispatch(self.controller, action,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1391, in dispatch
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return method(*args,
**kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py",
line 416, in wrapped
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return func(self, req,
*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 298, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._restore(image_repo, image)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 163, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi image.set_data(data,
size, backend=backend)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py",
line 208, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.base.set_data(data, size, backend=backend, set_active=set_active)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 501, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
_send_notification(notify_error, 'image.upload', msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_,
self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 447, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.repo.set_data(data, size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py",
line 198, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
self.image.set_data(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py",
line 318, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self.image.set_data(data, size=size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 567, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
self._upload_to_store(data, verifier, backend, size)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 458, in _upload_to_store
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi multihash, loc_meta) =
self.store_api.add_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 398, in add_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
store_add_to_backend_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 480, in store_add_to_backend_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi (location, size,
checksum, multihash, metadata) = store.add(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py",
line 279, in add_adapter
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi metadata_dict) =
store_add_fun(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py",
line 176, in op_checker
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return
store_op_fun(store, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1082, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise
glance_store.BackendException(msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
glance_store.exceptions.BackendException: Failed to add object to Swift.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Got error from Swift:
put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...)
failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
I don't see errors in the swift logs:
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift
- - - - txf319964369254af4974a7-006062ec22 - 0.0135 RL - 1617095714.876815081
1617095714.890305758 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift
- - - - txf319964369254af4974a7-006062ec22 - 0.0131 CS - 1617095714.892540216
1617095714.905595303 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - -
30/Mar/2021/09/15/14 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting
HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0127 CS -
1617095714.907815695 1617095714.920493126 0
Mar 30 09:15:15 infra1-swift-proxy-container-27169fa7 proxy-server[1068]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/15/15 HEAD
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting
HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - -
txf319964369254af4974a7-006062ec22 - 0.2779 - - 1617095714.874005318
1617095715.151889324 0
Mar 30 09:15:17 infra1-swift-proxy-container-27169fa7 proxy-server[1071]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/15/17 PUT
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001
HTTP/1.0 201 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 - -
txc05fff07e3384a68a7fa9-006062ec23 - 2.8218 - - 1617095715.163244247
1617095717.985031843 0
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]:
Client disconnected without sending last chunk (txn:
tx49b0efe394364bf1975bb-006062ec25) (client_ip: 192.168.110.106)
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/16/10 PUT
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002
HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 89 -
tx49b0efe394364bf1975bb-006062ec25 - 52.2224 - - 1617095717.995234489
1617095770.217619896 0
Mar 30 09:16:11 infra1-swift-proxy-container-27169fa7 proxy-server[1080]:
192.168.110.106 192.168.110.211 30/Mar/2021/09/16/11 DELETE
/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001
HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - -
tx98a9247419a2445fafcbf-006062ec5b - 0.0486 - - 1617095771.441193104
1617095771.489775896 0
and
Mar 30 09:15:17 bc1bl12 object-server[694851]: 10.0.3.211 - -
[30/Mar/2021:09:15:17 +0000] "PUT
/os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"
201 - "PUT
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001";
"txc05fff07e3384a68a7fa9-006062ec23" "proxy-server 1071" 2.8030 "-" 694851 0
Mar 30 09:16:10 bc1bl12 object-server[694850]: 10.0.3.211 - -
[30/Mar/2021:09:16:10 +0000] "PUT
/os-objects/43/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002"
499 89 "PUT
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002";
"tx49b0efe394364bf1975bb-006062ec25" "proxy-server 1067" 52.1551 "-" 694850 0
Mar 30 09:16:11 bc1bl12 object-server[694854]: 10.0.3.211 - -
[30/Mar/2021:09:16:11 +0000] "DELETE
/os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"
204 - "DELETE
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001";
"tx98a9247419a2445fafcbf-006062ec5b" "proxy-server 1080" 0.0321 "-" 694854 0
nova-compute shows the following:
Mar 30 09:13:24 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:24.975 650019
INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] instance snapshotting
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.038 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Beginning live
snapshot process
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.527 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Skipping quiescing
instance: QEMU guest agent is not enabled.
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.384 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.386 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.983 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.390 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.391 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:14:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:58.006 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:15:13 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:13.861 650019
INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Snapshot extracted,
beginning image upload
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.395 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.397 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.997 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.530 650019
INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted
task state from image_uploading on failure for instance.
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server [req-551ffa80-a1c6-462d-931a-84069b23995a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Exception during message handling:
glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The
server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2478, in snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server metadata['location'] =
root_disk.direct_snapshot(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py",
line 452, in direct_snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise
NotImplementedError(_('direct_snapshot() is not implemented'))
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not
implemented
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server During handling of the above exception, another
exception occurred:
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method,
ctxt, args)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 76, in wrapped
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 69, in wrapped
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 188, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state
for instance. "
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 159, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py",
line 1434, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 216, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 205, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 236, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server compute_utils.delete_image(
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 232, in decorated_function
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return function(self, context, image_id,
instance,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3908, in snapshot_instance
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id,
instance,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3941, in _snapshot_instance
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance,
image_id,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2549, in snapshot
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server self._image_api.update(context,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1247, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return session.update(context, image_id,
image_info, data=data,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 696, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1035, in _reraise_translated_image_exception
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 694, in update
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server image = self._update_v2(context,
sent_service_image_meta, data)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 713, in _update_v2
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server image = self._upload_data(context,
image_id, data)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 589, in _upload_data
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line
694, in tpool_execute
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 192, in call
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args,
**kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py",
line 600, in inner
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args,
**kwargs))
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py",
line 289, in upload
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url,
headers=hdrs, data=body)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py",
line 404, in put
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 380, in request
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server return self._handle_response(resp)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 120, in _handle_response
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation.
2021-03-30 09:16:12.533 650019
ERROR oslo_messaging.rpc.server
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.391 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.392 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.997 650019
INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - -
- - -] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1921893/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
