Public bug reported: On a large number of instances 'nova list' may return 404, probably this is because some instances are deleted during command execution. Trace:
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] Caught error: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found. 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack Traceback (most recent call last): 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 139, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return req.get_response(self.application) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack application, catch_exc_info=False) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack response = req.get_response(self._app) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack application, catch_exc_info=False) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack response = self.app(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 672, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack content_type, body, accept) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack request, action_args) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack **action_args) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/extended_server_attributes.py", line 97, in detail 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack instances.values()) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3597, in get_instances_host_statuses 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack host_status = self.get_instance_host_status(instance) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3575, in get_instance_host_status 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack service = [service for service in instance.services if 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack self.obj_load_attr(name) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 925, in obj_load_attr 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack self._load_generic(attrname) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 753, in _load_generic 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack expected_attrs=[attrname]) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack result = fn(cls, context, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 399, in get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack use_slave=use_slave) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 285, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(*args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack columns_to_join=columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 696, in instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(*args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 330, in wrapped 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(context, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1967, in instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack columns_to_join=columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1976, in _instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack raise exception.InstanceNotFound(instance_id=uuid) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack InstanceNotFound: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found. 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack 2016-08-05 09:30:52.672 878 INFO nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] http://192.168.0.2:8774/v2.1/servers/detail?all_tenants=1 returned with HTTP 404 2016-08-05 09:30:52.679 878 INFO nova.osapi_compute.wsgi.server [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] 192.168.0.2 "GET /v2.1/servers/detail?all_tenants=1 HTTP/1.1" status: 404 len: 363 time: 7.7833738 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1610153 Title: nova list can sometimes return 404 Status in OpenStack Compute (nova): New Bug description: On a large number of instances 'nova list' may return 404, probably this is because some instances are deleted during command execution. Trace: 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] Caught error: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found. 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack Traceback (most recent call last): 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 139, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return req.get_response(self.application) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack application, catch_exc_info=False) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack response = req.get_response(self._app) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack application, catch_exc_info=False) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack response = self.app(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return resp(environ, start_response) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 672, in __call__ 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack content_type, body, accept) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack request, action_args) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack **action_args) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/extended_server_attributes.py", line 97, in detail 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack instances.values()) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3597, in get_instances_host_statuses 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack host_status = self.get_instance_host_status(instance) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3575, in get_instance_host_status 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack service = [service for service in instance.services if 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack self.obj_load_attr(name) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 925, in obj_load_attr 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack self._load_generic(attrname) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 753, in _load_generic 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack expected_attrs=[attrname]) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack result = fn(cls, context, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 399, in get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack use_slave=use_slave) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 285, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(*args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack columns_to_join=columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 696, in instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(*args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 330, in wrapped 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack return f(context, *args, **kwargs) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1967, in instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack columns_to_join=columns_to_join) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1976, in _instance_get_by_uuid 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack raise exception.InstanceNotFound(instance_id=uuid) 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack InstanceNotFound: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found. 2016-08-05 09:30:52.666 878 ERROR nova.api.openstack 2016-08-05 09:30:52.672 878 INFO nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] http://192.168.0.2:8774/v2.1/servers/detail?all_tenants=1 returned with HTTP 404 2016-08-05 09:30:52.679 878 INFO nova.osapi_compute.wsgi.server [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] 192.168.0.2 "GET /v2.1/servers/detail?all_tenants=1 HTTP/1.1" status: 404 len: 363 time: 7.7833738 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1610153/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
