Public bug reported: Rally job is failing in the gate due to the following error during cleanup [0]:
2017-03-03 13:14:56.897549 | 2017-03-03 13:14:56.897 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete) 2017-03-03 13:14:56.899015 | 2017-03-03 13:14:56.898 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/common/utils.py", line 223, in retry 2017-03-03 13:14:56.900375 | 2017-03-03 13:14:56.900 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs) 2017-03-03 13:14:56.901935 | 2017-03-03 13:14:56.901 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/plugins/openstack/cleanup/resources.py", line 472, in delete 2017-03-03 13:14:56.903254 | 2017-03-03 13:14:56.902 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete_quota(self.tenant_uuid) 2017-03-03 13:14:56.904746 | 2017-03-03 13:14:56.904 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator 2017-03-03 13:14:56.906156 | 2017-03-03 13:14:56.905 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs) 2017-03-03 13:14:56.907779 | 2017-03-03 13:14:56.907 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in delete_quota 2017-03-03 13:14:56.909166 | 2017-03-03 13:14:56.908 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return self.delete(self.quota_path % (project_id)) 2017-03-03 13:14:56.910455 | 2017-03-03 13:14:56.910 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in delete 2017-03-03 13:14:56.911722 | 2017-03-03 13:14:56.911 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.913068 | 2017-03-03 13:14:56.912 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request 2017-03-03 13:14:56.914760 | 2017-03-03 13:14:56.914 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.916094 | 2017-03-03 13:14:56.915 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 301, in do_request 2017-03-03 13:14:56.917732 | 2017-03-03 13:14:56.917 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._handle_fault_response(status_code, replybody, resp) 2017-03-03 13:14:56.919266 | 2017-03-03 13:14:56.918 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 276, in _handle_fault_response 2017-03-03 13:14:56.920699 | 2017-03-03 13:14:56.920 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager exception_handler_v20(status_code, error_body) 2017-03-03 13:14:56.922342 | 2017-03-03 13:14:56.922 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 2017-03-03 13:14:56.923773 | 2017-03-03 13:14:56.923 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager request_ids=request_ids) 2017-03-03 13:14:56.925720 | 2017-03-03 13:14:56.925 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager NotFound: Quota for tenant 82ee0ba1b6534f958d1acd2f717b5c3d could not be found. Seems like we're hitting this errors from 1AM today (03/03/2017) [1] [0] http://logs.openstack.org/91/431691/30/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/ab3471c/console.html#_2017-03-03_13_14_56_925720 [1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20%5C%22gate-rally-dsvm-neutron-neutron-ubuntu-xenial%5C%22%20AND%20message%3A%20%5C%22Quota%20for%20tenant%5C%22 ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1669805 Title: rally job failing in gate due to "Quota for tenant X could not be found" Error Status in neutron: New Bug description: Rally job is failing in the gate due to the following error during cleanup [0]: 2017-03-03 13:14:56.897549 | 2017-03-03 13:14:56.897 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete) 2017-03-03 13:14:56.899015 | 2017-03-03 13:14:56.898 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/common/utils.py", line 223, in retry 2017-03-03 13:14:56.900375 | 2017-03-03 13:14:56.900 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs) 2017-03-03 13:14:56.901935 | 2017-03-03 13:14:56.901 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/plugins/openstack/cleanup/resources.py", line 472, in delete 2017-03-03 13:14:56.903254 | 2017-03-03 13:14:56.902 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete_quota(self.tenant_uuid) 2017-03-03 13:14:56.904746 | 2017-03-03 13:14:56.904 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator 2017-03-03 13:14:56.906156 | 2017-03-03 13:14:56.905 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs) 2017-03-03 13:14:56.907779 | 2017-03-03 13:14:56.907 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in delete_quota 2017-03-03 13:14:56.909166 | 2017-03-03 13:14:56.908 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return self.delete(self.quota_path % (project_id)) 2017-03-03 13:14:56.910455 | 2017-03-03 13:14:56.910 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in delete 2017-03-03 13:14:56.911722 | 2017-03-03 13:14:56.911 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.913068 | 2017-03-03 13:14:56.912 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request 2017-03-03 13:14:56.914760 | 2017-03-03 13:14:56.914 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.916094 | 2017-03-03 13:14:56.915 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 301, in do_request 2017-03-03 13:14:56.917732 | 2017-03-03 13:14:56.917 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._handle_fault_response(status_code, replybody, resp) 2017-03-03 13:14:56.919266 | 2017-03-03 13:14:56.918 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 276, in _handle_fault_response 2017-03-03 13:14:56.920699 | 2017-03-03 13:14:56.920 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager exception_handler_v20(status_code, error_body) 2017-03-03 13:14:56.922342 | 2017-03-03 13:14:56.922 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 2017-03-03 13:14:56.923773 | 2017-03-03 13:14:56.923 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager request_ids=request_ids) 2017-03-03 13:14:56.925720 | 2017-03-03 13:14:56.925 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager NotFound: Quota for tenant 82ee0ba1b6534f958d1acd2f717b5c3d could not be found. Seems like we're hitting this errors from 1AM today (03/03/2017) [1] [0] http://logs.openstack.org/91/431691/30/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/ab3471c/console.html#_2017-03-03_13_14_56_925720 [1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20%5C%22gate-rally-dsvm-neutron-neutron-ubuntu-xenial%5C%22%20AND%20message%3A%20%5C%22Quota%20for%20tenant%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1669805/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
