Reviewed: https://review.opendev.org/c/openstack/nova/+/803322 Committed: https://opendev.org/openstack/nova/commit/dbf3e276ba5971f1f6483e04b84cb0fc403cd592 Submitter: "Zuul (22348)" Branch: master
commit dbf3e276ba5971f1f6483e04b84cb0fc403cd592 Author: Lee Yarwood <[email protected]> Date: Tue Aug 3 10:05:42 2021 +0100 zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm As seen before in bug #1913451 the nova-lvm job creates large RAW snapshots during each run due to the nature of using LVM storage for the ephemeral instance disks. Id425aa546f1a5973bae8be9c017782d18f0b4a47 recently landed capping the per teneant image size total to 1000 MiB that is far too low for this job given the RAW snapshots it's creating. This in turn resulted in 504 failures when n-cpu attempted to upload snapshots to g-api. This change simply increases the per tenant limit to 10 GiB that should be more than enough for all of the tests covered by the nova-lvm job. Closes-Bug: #1938765 Change-Id: I824655387a10ac9b813c1b7b2399e25ed95f7fc3 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1938765 Title: nova-lvm job constantly fails on Glance image upload Status in OpenStack Compute (nova): Fix Released Bug description: The nova-lvm job seems to fail for every change [1] which prevents us to merge any change touching nova/virt/libvirt [2] All failures seem to relate to the same Tempest tests (6 of them) failing with the same problem : a Glance image upload issue as Glance API returns a HTTP502. Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest- ListImageFiltersTestJSON-1499681503 tempest- ListImageFiltersTestJSON-1499681503-project] Exception during message handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80 Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot( Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented')) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist- packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist- packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification( Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise() Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise self.value Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 183, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise() Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise self.value Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 154, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1433, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 211, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise() Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise self.value Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 231, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server compute_utils.delete_image( Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise() Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise self.value Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 227, in decorated_function Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance, Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3917, in snapshot_instance Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance, Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3950, in _snapshot_instance Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id, Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server self._image_api.update(context, Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 1240, in update Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data, Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 690, in update Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 1028, in _reraise_translated_image_exception Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 688, in update Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2 Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call, Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/utils.py", line 687, in tpool_execute Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise value Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 190, in call Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py", line 628, in inner Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs)) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py", line 308, in upload Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 404, in put Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py", line 380, in request Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server return self._handle_response(resp) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py", line 120, in _handle_response Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content) Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80 Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova- compute[106038]: ERROR oslo_messaging.rpc.server [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master [2] https://zuul.openstack.org/job/nova-lvm To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1938765/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
