Public bug reported: Seen here:
http://logs.openstack.org/59/424759/12/gate/gate-tempest-dsvm-cells- ubuntu-xenial/d7b1311/console.html#_2017-01-31_17_48_34_663273 2017-01-31 17:48:34.663337 | Captured traceback: 2017-01-31 17:48:34.663348 | ~~~~~~~~~~~~~~~~~~~ 2017-01-31 17:48:34.663363 | Traceback (most recent call last): 2017-01-31 17:48:34.663393 | File "tempest/api/compute/admin/test_servers.py", line 59, in test_list_servers_filter_by_error_status 2017-01-31 17:48:34.663414 | self.assertIn(self.s1_id, map(lambda x: x['id'], servers)) 2017-01-31 17:48:34.663448 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn 2017-01-31 17:48:34.663468 | self.assertThat(haystack, Contains(needle), message) 2017-01-31 17:48:34.663502 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2017-01-31 17:48:34.663515 | raise mismatch_error 2017-01-31 17:48:34.663542 | testtools.matchers._impl.MismatchError: u'108b4797-74fd-4a00-912a-b7fe0e142888' not in [] This test resets the state on a server to ERROR: 2017-01-31 17:48:34.663649 | 2017-01-31 17:28:39,375 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 202 POST http://10.23.154.32:8774/v2.1/servers/108b4797-74fd-4a00-912a-b7fe0e142888/action 0.142s 2017-01-31 17:48:34.663695 | 2017-01-31 17:28:39,376 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'} 2017-01-31 17:48:34.663714 | Body: {"os-resetState": {"state": "error"}} Then tries to list servers by that status and expects to get that one back: 2017-01-31 17:48:34.663883 | 2017-01-31 17:28:39,556 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 200 GET http://10.23.154.32:8774/v2.1/servers?status=error 0.179s 2017-01-31 17:48:34.663955 | 2017-01-31 17:28:39,556 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'} 2017-01-31 17:48:34.663969 | Body: None 2017-01-31 17:48:34.664078 | Response - Headers: {u'x-openstack-nova-api-version': '2.1', u'vary': 'X-OpenStack-Nova-API-Version', u'content-length': '15', 'status': '200', u'content-type': 'application/json', u'x-compute-request-id': 'req-91ef16ab-28c3-47c5-b823-6a321bde5c01', u'date': 'Tue, 31 Jan 2017 17:28:39 GMT', 'content-location': 'http://10.23.154.32:8774/v2.1/servers?status=error', u'openstack-api-version': 'compute 2.1', u'connection': 'close'} 2017-01-31 17:48:34.664094 | Body: {"servers": []} And the list is coming back empty, intermittently, with cells v1. So there is probably some vm_state change race between the state change in the child cell and reporting that back up to the parent API cell. http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Body%3A%20%7B%5C%5C%5C%22servers%5C%5C%5C%22%3A%20%5B%5D%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C %22gate-tempest-dsvm-cells-ubuntu-xenial%5C%22&from=7d 16 hits in 7 days, check and gate, all failures. ** Affects: nova Importance: Medium Status: Confirmed ** Tags: cells testing v1 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1660747 Title: test_list_servers_filter_by_error_status intermittently fails with MismatchError on no servers in response Status in OpenStack Compute (nova): Confirmed Bug description: Seen here: http://logs.openstack.org/59/424759/12/gate/gate-tempest-dsvm-cells- ubuntu-xenial/d7b1311/console.html#_2017-01-31_17_48_34_663273 2017-01-31 17:48:34.663337 | Captured traceback: 2017-01-31 17:48:34.663348 | ~~~~~~~~~~~~~~~~~~~ 2017-01-31 17:48:34.663363 | Traceback (most recent call last): 2017-01-31 17:48:34.663393 | File "tempest/api/compute/admin/test_servers.py", line 59, in test_list_servers_filter_by_error_status 2017-01-31 17:48:34.663414 | self.assertIn(self.s1_id, map(lambda x: x['id'], servers)) 2017-01-31 17:48:34.663448 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn 2017-01-31 17:48:34.663468 | self.assertThat(haystack, Contains(needle), message) 2017-01-31 17:48:34.663502 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2017-01-31 17:48:34.663515 | raise mismatch_error 2017-01-31 17:48:34.663542 | testtools.matchers._impl.MismatchError: u'108b4797-74fd-4a00-912a-b7fe0e142888' not in [] This test resets the state on a server to ERROR: 2017-01-31 17:48:34.663649 | 2017-01-31 17:28:39,375 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 202 POST http://10.23.154.32:8774/v2.1/servers/108b4797-74fd-4a00-912a-b7fe0e142888/action 0.142s 2017-01-31 17:48:34.663695 | 2017-01-31 17:28:39,376 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'} 2017-01-31 17:48:34.663714 | Body: {"os-resetState": {"state": "error"}} Then tries to list servers by that status and expects to get that one back: 2017-01-31 17:48:34.663883 | 2017-01-31 17:28:39,556 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 200 GET http://10.23.154.32:8774/v2.1/servers?status=error 0.179s 2017-01-31 17:48:34.663955 | 2017-01-31 17:28:39,556 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'} 2017-01-31 17:48:34.663969 | Body: None 2017-01-31 17:48:34.664078 | Response - Headers: {u'x-openstack-nova-api-version': '2.1', u'vary': 'X-OpenStack-Nova-API-Version', u'content-length': '15', 'status': '200', u'content-type': 'application/json', u'x-compute-request-id': 'req-91ef16ab-28c3-47c5-b823-6a321bde5c01', u'date': 'Tue, 31 Jan 2017 17:28:39 GMT', 'content-location': 'http://10.23.154.32:8774/v2.1/servers?status=error', u'openstack-api-version': 'compute 2.1', u'connection': 'close'} 2017-01-31 17:48:34.664094 | Body: {"servers": []} And the list is coming back empty, intermittently, with cells v1. So there is probably some vm_state change race between the state change in the child cell and reporting that back up to the parent API cell. http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Body%3A%20%7B%5C%5C%5C%22servers%5C%5C%5C%22%3A%20%5B%5D%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C %22gate-tempest-dsvm-cells-ubuntu-xenial%5C%22&from=7d 16 hits in 7 days, check and gate, all failures. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1660747/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
