Public bug reported:
The nova-lvm job seems to fail for every change [1] which prevents us to
merge any change touching nova/virt/libvirt [2]
All failures seem to relate to the same Tempest tests (6 of them)
failing with the same problem : a Glance image upload issue as Glance
API returns a HTTP502.
Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]:
ERROR os_brick.initiator.linuxscsi [None
req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c
tempest-VolumesAdminNegativeTest-243853426
tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running:
exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2]
No such file or directory
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server [None
req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
ListImageFiltersTestJSON-1499681503 tempest-
ListImageFiltersTestJSON-1499681503-project] Exception during message
handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
Gateway: The proxy server received an invalid: response from an upstream
server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
metadata['location'] = root_disk.direct_snapshot(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
direct_snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
NotImplementedError(_('direct_snapshot() is not implemented'))
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
direct_snapshot() is not implemented
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server During handling of the
above exception, another exception occurred:
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result = func(ctxt,
**new_args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 183, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server LOG.warning("Failed
to revert task state for instance. "
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 154, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/utils.py", line 1433, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 211, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 200, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 231, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.delete_image(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 227, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3917, in
snapshot_instance
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._snapshot_instance(context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3950, in
_snapshot_instance
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.driver.snapshot(context, instance, image_id,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._image_api.update(context,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1240, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
session.update(context, image_id, image_info, data=data,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 690, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1028, in
_reraise_translated_image_exception
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
new_exc.with_traceback(exc_trace)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 688, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._update_v2(context, sent_service_image_meta, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._upload_data(context, image_id, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
utils.tpool_execute(self._client.call,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/utils.py", line 687, in tpool_execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
eventlet.tpool.execute(func, *args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in
execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server six.reraise(c, e,
tb)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise value
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in
tworker
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server rv = meth(*args,
**kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 190, in call
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result =
getattr(controller, method)(*args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
line 628, in inner
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
RequestIdProxy(wrapped(*args, **kwargs))
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py", line
308, in upload
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server resp, body =
self.http_client.put(url, headers=hdrs, data=body)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line
404, in put
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self.request(url, 'PUT', **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 380, in request
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._handle_response(resp)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 120, in _handle_response
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
exc.from_response(resp, resp.content)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The
proxy server received an invalid: response from an upstream server.:
Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
[2] https://zuul.openstack.org/job/nova-lvm
** Affects: nova
Importance: Critical
Status: Confirmed
** Tags: gate-failure
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1938765
Title:
nova-lvm job constantly fails on Glance image upload
Status in OpenStack Compute (nova):
Confirmed
Bug description:
The nova-lvm job seems to fail for every change [1] which prevents us
to merge any change touching nova/virt/libvirt [2]
All failures seem to relate to the same Tempest tests (6 of them)
failing with the same problem : a Glance image upload issue as Glance
API returns a HTTP502.
Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]:
ERROR os_brick.initiator.linuxscsi [None
req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c
tempest-VolumesAdminNegativeTest-243853426
tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running:
exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2]
No such file or directory
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server [None
req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
ListImageFiltersTestJSON-1499681503 tempest-
ListImageFiltersTestJSON-1499681503-project] Exception during message
handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
Gateway: The proxy server received an invalid: response from an
upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port
80
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
metadata['location'] = root_disk.direct_snapshot(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
direct_snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
NotImplementedError(_('direct_snapshot() is not implemented'))
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
direct_snapshot() is not implemented
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server During handling of
the above exception, another exception occurred:
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 183, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
LOG.warning("Failed to revert task state for instance. "
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 154, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/utils.py", line 1433, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 211, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 200, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 231, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.delete_image(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 227, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3917, in
snapshot_instance
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._snapshot_instance(context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3950, in
_snapshot_instance
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.driver.snapshot(context, instance, image_id,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._image_api.update(context,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1240, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
session.update(context, image_id, image_info, data=data,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 690, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1028, in
_reraise_translated_image_exception
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
new_exc.with_traceback(exc_trace)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 688, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._update_v2(context, sent_service_image_meta, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._upload_data(context, image_id, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
utils.tpool_execute(self._client.call,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/utils.py", line 687, in tpool_execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
eventlet.tpool.execute(func, *args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
in execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server six.reraise(c, e,
tb)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise value
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
in tworker
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server rv = meth(*args,
**kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 190, in call
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result =
getattr(controller, method)(*args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
line 628, in inner
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
RequestIdProxy(wrapped(*args, **kwargs))
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py",
line 308, in upload
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server resp, body =
self.http_client.put(url, headers=hdrs, data=body)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py",
line 404, in put
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self.request(url, 'PUT', **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 380, in request
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._handle_response(resp)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 120, in _handle_response
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
exc.from_response(resp, resp.content)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway:
The proxy server received an invalid: response from an upstream
server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
[2] https://zuul.openstack.org/job/nova-lvm
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1938765/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
