[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Changed in: nova/train Status: New => Fix Released ** Changed in: nova/train Importance: Undecided => Low ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/rocky Importance: Undecided => Low ** Changed in: nova/stein Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1783565 Title: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was up." Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: Fix Released Bug description: http://logs.openstack.org/32/584032/5/check/nova-tox-functional- py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415 18-07-25 03:16:46.418499 | ubuntu-xenial | {5} nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host [14.070214s] ... FAILED 2018-07-25 03:16:46.418582 | ubuntu-xenial | 2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback: 2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~ 2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent call last):' 2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py", line 456, in test_evacuate_with_anti_affinity_no_valid_host' 2018-07-25 03:16:46.419232 | ubuntu-xenial | b" self.admin_api.post_server_action(servers[1]['id'], post)" 2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 294, in post_server_action' 2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' % server_id, data, **kwargs).body" 2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 235, in api_post' 2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return APIResponse(self.api_request(relative_uri, **kwargs))' 2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 213, in api_request' 2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)' 2018-07-25 03:16:46.420545 | ubuntu-xenial | b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"badRequest": {"message": "Compute service of host2 is still in use.", "code": 400}}' 2018-07-25 03:16:46.420581 | ubuntu-xenial | b'' 2018-07-25 03:16:46.420606 | ubuntu-xenial | 2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr: 2018-07-25 03:16:46.420702 | ubuntu-xenial | 2018-07-25 03:16:46.421102 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.421623 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.422054 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'" 2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture = self.useFixture(moxstubout.MoxStubout())' 2018-07-25 03:16:46.422537 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.' 2018-07-25 03:16:46.422664 | ubuntu-xenial | b' return pkg_resources.EntryPoint.parse("x=" + s).load(False)' 2018-07-25 03:16:46.422928 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/opensta
[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
Reviewed: https://review.opendev.org/692252 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c93ca82b8fb8e39787020decfe335d45f8732e7 Submitter: Zuul Branch:master commit 1c93ca82b8fb8e39787020decfe335d45f8732e7 Author: melanie witt Date: Thu Oct 31 01:57:41 2019 + Replace time.sleep(10) with service forced_down in tests The server group functional tests are doing time.sleep(10) in order to make sure a stopped compute service is considered "down" by the nova compute API. Instead of sleeping, we can set the service as "forced_down" to get the desired "down" compute service status and avoid unnecessary delays in these tests. Unnecessary service start() calls are also removed in this change. They appear at the end of tests and services are started during each test setUp() and killed during each test tearDown() via the ServiceFixture. Closes-Bug: #1783565 Change-Id: I74f64b68e4b33ee0f8c45fdc5f570c7e12e05d3b ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1783565 Title: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was up." Status in OpenStack Compute (nova): Fix Released Bug description: http://logs.openstack.org/32/584032/5/check/nova-tox-functional- py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415 18-07-25 03:16:46.418499 | ubuntu-xenial | {5} nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host [14.070214s] ... FAILED 2018-07-25 03:16:46.418582 | ubuntu-xenial | 2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback: 2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~ 2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent call last):' 2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py", line 456, in test_evacuate_with_anti_affinity_no_valid_host' 2018-07-25 03:16:46.419232 | ubuntu-xenial | b" self.admin_api.post_server_action(servers[1]['id'], post)" 2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 294, in post_server_action' 2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' % server_id, data, **kwargs).body" 2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 235, in api_post' 2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return APIResponse(self.api_request(relative_uri, **kwargs))' 2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 213, in api_request' 2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)' 2018-07-25 03:16:46.420545 | ubuntu-xenial | b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"badRequest": {"message": "Compute service of host2 is still in use.", "code": 400}}' 2018-07-25 03:16:46.420581 | ubuntu-xenial | b'' 2018-07-25 03:16:46.420606 | ubuntu-xenial | 2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr: 2018-07-25 03:16:46.420702 | ubuntu-xenial | 2018-07-25 03:16:46.421102 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.421623 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.422054 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'" 2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture = self.useFixture(moxstubout.MoxStubout())' 2018-07-25 03:16:46.422537 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack
[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
gibi has a fix here: https://review.opendev.org/692252 ** Changed in: nova Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer) ** Changed in: nova Status: Invalid => In Progress ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1783565 Title: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was up." Status in OpenStack Compute (nova): In Progress Bug description: http://logs.openstack.org/32/584032/5/check/nova-tox-functional- py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415 18-07-25 03:16:46.418499 | ubuntu-xenial | {5} nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host [14.070214s] ... FAILED 2018-07-25 03:16:46.418582 | ubuntu-xenial | 2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback: 2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~ 2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent call last):' 2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py", line 456, in test_evacuate_with_anti_affinity_no_valid_host' 2018-07-25 03:16:46.419232 | ubuntu-xenial | b" self.admin_api.post_server_action(servers[1]['id'], post)" 2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 294, in post_server_action' 2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' % server_id, data, **kwargs).body" 2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 235, in api_post' 2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return APIResponse(self.api_request(relative_uri, **kwargs))' 2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 213, in api_request' 2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)' 2018-07-25 03:16:46.420545 | ubuntu-xenial | b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"badRequest": {"message": "Compute service of host2 is still in use.", "code": 400}}' 2018-07-25 03:16:46.420581 | ubuntu-xenial | b'' 2018-07-25 03:16:46.420606 | ubuntu-xenial | 2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr: 2018-07-25 03:16:46.420702 | ubuntu-xenial | 2018-07-25 03:16:46.421102 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.421623 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.422054 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'" 2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture = self.useFixture(moxstubout.MoxStubout())' 2018-07-25 03:16:46.422537 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.' 2018-07-25 03:16:46.422664 | ubuntu-xenial | b' return pkg_resources.EntryPoint.parse("x=" + s).load(False)' 2018-07-25 03:16:46.422928 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/api.py:205: DeprecationWarning: Property 'async_compat' has moved to 'function.async_'" 2018-07-25 03:16:46.423038 | ubuntu-xenial | b' reader_mode = get_context_manager(context).async' 2018-07-25 03:16:46.423301 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/api.py:205: DeprecationWarning: Property 'async_compat' has moved to 'function.async_'" 2018-07-25 0
[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
We don't seem to be hitting this in the gate anymore so I'm not sure if it's just rare now or if it's resolved some other way: http://status.openstack.org/elastic-recheck/#1783565 I'm marking invalid for now though. We can re-open if necessary. ** Changed in: nova Assignee: Zhenyu Zheng (zhengzhenyu) => (unassigned) ** Changed in: nova Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1783565 Title: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was up." Status in OpenStack Compute (nova): Invalid Bug description: http://logs.openstack.org/32/584032/5/check/nova-tox-functional- py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415 18-07-25 03:16:46.418499 | ubuntu-xenial | {5} nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host [14.070214s] ... FAILED 2018-07-25 03:16:46.418582 | ubuntu-xenial | 2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback: 2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~ 2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent call last):' 2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py", line 456, in test_evacuate_with_anti_affinity_no_valid_host' 2018-07-25 03:16:46.419232 | ubuntu-xenial | b" self.admin_api.post_server_action(servers[1]['id'], post)" 2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 294, in post_server_action' 2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' % server_id, data, **kwargs).body" 2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 235, in api_post' 2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return APIResponse(self.api_request(relative_uri, **kwargs))' 2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py", line 213, in api_request' 2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)' 2018-07-25 03:16:46.420545 | ubuntu-xenial | b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"badRequest": {"message": "Compute service of host2 is still in use.", "code": 400}}' 2018-07-25 03:16:46.420581 | ubuntu-xenial | b'' 2018-07-25 03:16:46.420606 | ubuntu-xenial | 2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr: 2018-07-25 03:16:46.420702 | ubuntu-xenial | 2018-07-25 03:16:46.421102 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.421623 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade' 2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade = LegacyEngineFacade(None, _factory=self)' 2018-07-25 03:16:46.422054 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'" 2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture = self.useFixture(moxstubout.MoxStubout())' 2018-07-25 03:16:46.422537 | ubuntu-xenial | b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.' 2018-07-25 03:16:46.422664 | ubuntu-xenial | b' return pkg_resources.EntryPoint.parse("x=" + s).load(False)' 2018-07-25 03:16:46.422928 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/api.py:205: DeprecationWarning: Property 'async_compat' has moved to 'function.async_'" 2018-07-25 03:16:46.423038 | ubuntu-xenial | b' reader_mode = get_context_manager(context).async' 2018-07-25 03:16:46.423301 | ubuntu-xenial | b"/home/zuul/src/git.openstack.org/opensta