Public bug reported: Description =========== During 9.0RC2 Certification Testing Sutie run, the following error log appeared when executing "NovaSecGroup.boot_and_delete_server_with_secgroups" task:
2016-06-28 01:58:17.619 30929 DEBUG novaclient.v2.client [-] GET call to compute for http://172.16.184.3:8774/v2.1/servers/e3a9f1fd-d7a9-43af-a911-903297545f4e used request id req-33956e34-3b21-4f97-a0ae-fc975f9cc6b5 _log_request_id /opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py:85 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner [-] Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status. Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'} 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Traceback (most recent call last): 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner deprecated_output = getattr(scenario_inst, method_name)(**kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 137, in boot_and_delete_server_with_secgroups 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner f = func(self, *args, **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 147, in _boot_server 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner check_interval=CONF.benchmark.nova_server_boot_poll_interval 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner return f(*args, **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 147, in wait_for 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner check_interval=check_interval) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner resource = update_resource(resource) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 90, in _get_from_manager 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner fault=getattr(res, "fault", "n/a")) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner GetResourceErrorStatus: Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status. 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'} Steps to reproduce ================== Install Rally on masternode and run task "NovaSecGroup.boot_and_delete_server_with_secgroups": 1. cd /opt/stack 2. source .venv/bin/activate 3. rally task start /opt/stack/rally-scenarios/nova/boot_and_delete_server_with_secgroups.yaml Expected result =============== Task run without errors Actual result ============= Errors described above in logs Environment =========== Scale 200-node lab (ENV-10): 3 controllers 199 compute nodes 20 ceph nodes DVR enabled XVLAN Image RC2 build: fuel-9.0-mos-495-2016-06-16_18-18-00.iso Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz ** Affects: nova Importance: Undecided Status: New ** Attachment added: "Logs and YAML scenario" https://bugs.launchpad.net/bugs/1597848/+attachment/4692924/+files/attachment.tar.gz -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1597848 Title: Resource has ERROR status: Exceeded maximum number of retries Status in OpenStack Compute (nova): New Bug description: Description =========== During 9.0RC2 Certification Testing Sutie run, the following error log appeared when executing "NovaSecGroup.boot_and_delete_server_with_secgroups" task: 2016-06-28 01:58:17.619 30929 DEBUG novaclient.v2.client [-] GET call to compute for http://172.16.184.3:8774/v2.1/servers/e3a9f1fd-d7a9-43af-a911-903297545f4e used request id req-33956e34-3b21-4f97-a0ae-fc975f9cc6b5 _log_request_id /opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py:85 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner [-] Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status. Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'} 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Traceback (most recent call last): 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner deprecated_output = getattr(scenario_inst, method_name)(**kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 137, in boot_and_delete_server_with_secgroups 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner f = func(self, *args, **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 147, in _boot_server 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner check_interval=CONF.benchmark.nova_server_boot_poll_interval 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner return f(*args, **kwargs) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 147, in wait_for 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner check_interval=check_interval) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner resource = update_resource(resource) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 90, in _get_from_manager 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner fault=getattr(res, "fault", "n/a")) 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner GetResourceErrorStatus: Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status. 2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'} Steps to reproduce ================== Install Rally on masternode and run task "NovaSecGroup.boot_and_delete_server_with_secgroups": 1. cd /opt/stack 2. source .venv/bin/activate 3. rally task start /opt/stack/rally-scenarios/nova/boot_and_delete_server_with_secgroups.yaml Expected result =============== Task run without errors Actual result ============= Errors described above in logs Environment =========== Scale 200-node lab (ENV-10): 3 controllers 199 compute nodes 20 ceph nodes DVR enabled XVLAN Image RC2 build: fuel-9.0-mos-495-2016-06-16_18-18-00.iso Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1597848/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
