I'd though this was happening before it got to a compute host since no host was listed in the instances table of the database, but I did manage to hunt down the RPC exception that happens on the compute node:
2013-08-13 12:12:09.055 AUDIT nova.compute.manager [req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5 6f9adccbd03e4d2186756896957a14bf] [instance: 62f92c7a-a7cf-41e7-83c5-055ad43ad78a] Starting instance... 2013-08-13 12:12:11.279 ERROR nova.openstack.common.rpc.amqp [req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5 6f9adccbd03e4d2186756896957a14bf] Exception during message handling and then a long trace, included below. I suspected an incorrect rabbit password somewhere as the failure coincided with a config management update but I've mathc the passwords in config managenment and all teh openstack configs I can think of comparing both current values and backups from a known good state and haven't found that to be the case yet Thanks, -Jon TRACE: 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp temp_level, payload) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp pass 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1224, in run_instance 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp do_run_instance() 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1223, in do_run_instance 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 879, in _run_instance 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid']) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 800, in _run_instance 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image_meta = self._check_image_size(context, instance) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1006, in _check_image_size 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image_meta = _get_image_meta(context, instance['image_ref']) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 268, in _get_image_meta 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return image_service.show(context, image_id) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 239, in show 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp _reraise_translated_image_exception(image_id) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 237, in show 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image = self._client.call(context, 1, 'get', image_id) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 182, in call 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return getattr(client.images, method)(*args, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 104, in get 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp % urllib.quote(image_id)) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 260, in raw_request 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return self._http_request(url, method, **kwargs) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 221, in _http_request 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp raise exc.from_response(resp, body_str) 2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp ImageNotFound: Image 627b4902-f324-4615-a3bb-76f9fd22207a could not be found. On Mon, Aug 12, 2013 at 4:24 PM, John Bresnahan <[email protected]> wrote: > On 08/12/2013 09:09 AM, Jonathan Proulx wrote: > > > > This is particularly odd since glance was working fine yesterday and to > > my knowlege the only thing I did was turn on more compute nodes... > > > > > > Now when I try and launch an instance it goes almost immediately to > > error state with the fault: > > > > {u'message': u'ImageNotFound', u'code': 404, u'details': u'Image > > 627b4902-f324-4615-a3bb-76f9fd22207a could not be found.', u'created': > > u'2013-08-12T18:53:46Z'} > > > > I haven't yet found where the 404 is coming from, in debug mode the > > glance-registry.log shows: > > > > 2013-08-12 15:03:33.541 INFO glance.registry.api.v1.images > > [73bb53ea-be14-496b-9405-098ff8f432aa 0be8fa0d641a4e778b9262bd2e5f40b5 > > 6f9adccbd03e4d2186756896957a14bf] Successfully retrieved image > > 627b4902-f324-4615-a3bb-76f9fd22207a > > > > which looks like success to me, also the python-glanceclient will > > happily download the image. > > > > Help? > > Is there any thing in the nova compute log? > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
