Public bug reported: Description ===========
nova-lvm is often failing due to failures creating snapshots of instances during tempest runs: https://zuul.opendev.org/t/openstack/builds?job_name=nova- lvm&branch=master I *think* swift that is being used by g-api as the backing store is running out of space due to the nova-lvm job creating two file backed 24G volume groups, one for n-cpu and another for c-vol: https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-n-cpu.txt#37491 Jan 27 11:34:04.214271 ubuntu-focal-rax-ord-0022741499 nova- compute[90693]: ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 10.209.161.68 Port 80 https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-g-api.txt Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi [None req-a3f9e82f-5a4c-40fa-a102-7a1a3a7998a6 tempest-ServerActionsTestJSON-1412104246 tempest-ServerActionsTestJSON-1412104246-project] Caught error: Failed to add object to Swift. Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi Traceback (most recent call last): Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1014, in add Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi self._delete_stale_chunks( Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi self.force_reraise() Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi raise value Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1003, in add Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi manager.get_connection().put_object( Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 2015, in put_object Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1898, in _retry Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi reset_func(func, *args, **kwargs) Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1995, in _default_reset Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance', '0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005', ...) failure and no ability to reset contents for reupload. https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-s-proxy.txt?severity=0 Jan 27 11:34:01.340337 ubuntu-focal-rax-ord-0022741499 proxy-server[81160]: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: Object PUT exceptions during send, 0/1 required connections (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1913451 Title: nova-lvm Status in OpenStack Compute (nova): New Bug description: Description =========== nova-lvm is often failing due to failures creating snapshots of instances during tempest runs: https://zuul.opendev.org/t/openstack/builds?job_name=nova- lvm&branch=master I *think* swift that is being used by g-api as the backing store is running out of space due to the nova-lvm job creating two file backed 24G volume groups, one for n-cpu and another for c-vol: https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-n-cpu.txt#37491 Jan 27 11:34:04.214271 ubuntu-focal-rax-ord-0022741499 nova- compute[90693]: ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 10.209.161.68 Port 80 https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-g-api.txt Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi [None req-a3f9e82f-5a4c-40fa-a102-7a1a3a7998a6 tempest-ServerActionsTestJSON-1412104246 tempest-ServerActionsTestJSON-1412104246-project] Caught error: Failed to add object to Swift. Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi Traceback (most recent call last): Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1014, in add Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi self._delete_stale_chunks( Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi self.force_reraise() Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi raise value Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1003, in add Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi manager.get_connection().put_object( Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 2015, in put_object Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1898, in _retry Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi reset_func(func, *args, **kwargs) Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1995, in _default_reset Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance', '0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005', ...) failure and no ability to reset contents for reupload. https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-s-proxy.txt?severity=0 Jan 27 11:34:01.340337 ubuntu-focal-rax-ord-0022741499 proxy-server[81160]: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: Object PUT exceptions during send, 0/1 required connections (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1913451/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
