On Fri, Nov 1, 2013 at 5:45 PM, Dheerendra <[email protected]> wrote: > Hi Antonio > > I suspect the problem with rabbitMQ. Can you check it is up and running ?
RabbitMQ is up&running, but how can I check if it is *suffering*? The daemon is not using that much CPU, while nova-network is around 80-100% of the cpu when I start an instance I already increased the rpc_response_timeout value to 180 some times ago because it was failing when trying to start more than 15 instances at the same time, and it was working fine until a few days ago. > Can you look at the controller logs as well ? >From the server logs (in debug mode) I can't see anything if there is anything wrong. I don't see any "ERROR", and I see a lot of amqp messages that look all the same... I can provide the full logs if someone is willing to take a look at it :) .a. > > -Dheerendra > > > On Thu, Oct 31, 2013 at 10:58 PM, Antonio Messina > <[email protected]> wrote: >> >> Hi All, >> >> We are experiencing a problem with our OpenStack Folsom installation >> on Ubuntu 12.04, using nova-network: instances are NOT created, and >> deleting instances in error state fails. >> >> First of all: the installation was in "production" for almost one >> year, and we didn't change anything. The only change made was updating >> some packages from ubuntu and from ubuntu-cloud archives. >> >> (PS: The following text is formatted as restructured text) >> >> In details what happen is: >> >> * The instance is created `Thu Oct 31 18:01:00 CET 2013`:: >> >> 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi >> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] GET >> http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d >> 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi >> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] >> http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d >> returned with HTTP 200 >> 2013-10-31 18:01:02 INFO nova.osapi_compute.wsgi.server >> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] 130.60.24.9 - - [31/Oct/2013 18:01:02] >> "GET >> /v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d >> HTTP/1.1" 200 1531 0.225107 >> >> * On the compute node, I see multiple times the following trace:: >> >> 2013-10-31 18:01:26 931 INFO nova.compute.resource_tracker [-] >> Compute_service record updated for node-09-01-00 >> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free >> ram (MB): 17389 >> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free >> disk (GB): 136 >> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free >> VCPUS: 0 >> 2013-10-31 18:02:34 931 INFO nova.compute.resource_tracker [-] >> Compute_service record updated for node-09-01-00 >> 2013-10-31 18:02:41 931 ERROR nova.openstack.common.rpc.amqp [-] >> Exception during message handling >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> Traceback (most recent call last): >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 276, in _process_data >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp rval = >> self.proxy.dispatch(ctxt, version, method, **args) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", >> line 145, in dispatch >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return >> getattr(proxyobj, method)(ctxt, **kwargs) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> temp_level, payload) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return >> f(*args, **kw) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 390, in >> refresh_instance_security_rules >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return >> self.driver.refresh_instance_security_rules(instance) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2243, >> in refresh_instance_security_rules >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> self.firewall_driver.refresh_instance_security_rules(instance) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 446, in >> refresh_instance_security_rules >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> self.do_refresh_instance_rules(instance) >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 463, in >> do_refresh_instance_rules >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> network_info = self.network_infos[instance['id']] >> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp >> KeyError: 38331 >> >> This backtrace does not seem to be relevant, as I've already seen >> it in log of instances that started without problems. >> >> * After a while, IPs are assigned to the instance. I don't see >> anything in the ``nova-network.log``, but I can see the IPs in the >> ``nova list`` output. I also see the following errors in the compute >> node:: >> >> 2013-10-31 18:04:04 931 ERROR nova.openstack.common.rpc.common [-] >> Timed out waiting for RPC response: timed out >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> Traceback (most recent call last): >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 518, in ensure >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return method(*args, **kwargs) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 596, in _consume >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return self.connection.drain_events(timeout=timeout) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in >> drain_events >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return self.transport.drain_events(self.connection, **kwargs) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, >> in drain_events >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return connection.drain_events(**kwargs) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in >> drain_events >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return self.wait_multi(self.channels.values(), timeout=timeout) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in >> wait_multi >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> chanmap.keys(), allowed_methods, timeout=timeout) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, >> in _wait_multiple >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> channel, method_sig, args, content = read_timeout(timeout) >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in >> read_timeout >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> return self.method_reader.read_method() >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File >> "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", >> line 221, in read_method >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> raise m >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> timeout: timed out >> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common >> 2013-10-31 18:04:04 ERROR nova.compute.manager >> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] Instance failed network setup >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] Traceback (most recent call last): >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in >> _allocate_network >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] requested_networks=requested_networks) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] res = f(self, context, *args, >> **kwargs) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in >> allocate_for_instance >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] 'args': args}) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", >> line 108, in call >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] return _get_impl().call(cfg.CONF, >> context, topic, msg, timeout) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 718, in call >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] rpc_amqp.get_connection_pool(conf, >> Connection)) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 369, in call >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] rv = list(rv) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 330, in __iter__ >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] self.done() >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] self.gen.next() >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 327, in __iter__ >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] self._iterator.next() >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 601, in iterconsume >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] yield self.ensure(_error_callback, >> _consume) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 521, in ensure >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] error_callback(e) >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 582, in _error_callback >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] raise rpc_common.Timeout() >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] Timeout: Timeout while waiting on RPC >> response. >> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] >> 2013-10-31 18:04:04 INFO nova.compute.resource_tracker >> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] Aborting claim: [Claim >> ece6db13-10ee-4808-9646-57ab285ca33d: 14336 MB memory, 100 GB disk, 8 VCPUS] >> >> and then again:: >> >> 2013-10-31 18:07:04 ERROR nova.compute.manager >> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 >> 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: >> ece6db13-10ee-4808-9646-57ab285ca33d] Build error: ['Traceback (most recent >> call last):\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 498, in >> _run_instance\n requested_networks)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in >> _allocate_network\n requested_networks=requested_networks)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in >> wrapper\n res = f(self, context, *args, **kwargs)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in >> allocate_for_instance\n \'args\': args})\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", >> line 108, in call\n return _get_impl().call(cfg.CONF, context, topic, >> msg, timeout)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 718, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', ' >> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", >> line 369, in call\n rv = list(rv)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 330, in __iter__\n self.done()\n', ' File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n >> self.gen.next()\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 327, in __iter__\n self._iterator.next()\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 601, in iterconsume\n yield self.ensure(_error_callback, >> _consume)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 521, in ensure\n error_callback(e)\n', ' File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 582, in _error_callback\n raise rpc_common.Timeout()\n', 'Timeout: >> Timeout while waiting on RPC response.\n'] >> 2013-10-31 18:07:05 931 ERROR nova.openstack.common.rpc.amqp [-] >> Exception during message handling >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> Traceback (most recent call last): >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 276, in _process_data >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rval = >> self.proxy.dispatch(ctxt, version, method, **args) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", >> line 145, in dispatch >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return >> getattr(proxyobj, method)(ctxt, **kwargs) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> temp_level, payload) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return >> f(*args, **kw) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in >> decorated_function >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> pass >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in >> decorated_function >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return >> function(self, context, *args, **kwargs) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in >> decorated_function >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> kwargs['instance']['uuid'], e, sys.exc_info()) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in >> decorated_function >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return >> function(self, context, *args, **kwargs) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in >> run_instance >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> do_run_instance() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp retval >> = f(*args, **kwargs) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in >> do_run_instance >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> admin_password, is_first_time, instance) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in >> _run_instance >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self._set_instance_error_state(context, instance['uuid']) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in >> _run_instance >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> is_first_time, request_spec, filter_properties) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 547, in >> _reschedule_or_reraise >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self._deallocate_network(context, instance) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 783, in >> _deallocate_network >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.network_api.deallocate_for_instance(context, instance) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 261, in >> deallocate_for_instance >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp 'args': >> args}) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", >> line 108, in call >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return >> _get_impl().call(cfg.CONF, context, topic, msg, timeout) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 718, in call >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> rpc_amqp.get_connection_pool(conf, Connection)) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 369, in call >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rv = >> list(rv) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 330, in __iter__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.done() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self.gen.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line >> 327, in __iter__ >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> self._iterator.next() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 601, in iterconsume >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp yield >> self.ensure(_error_callback, _consume) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 521, in ensure >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> error_callback(e) >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", >> line 582, in _error_callback >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp raise >> rpc_common.Timeout() >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> Timeout: Timeout while waiting on RPC response. >> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp >> >> * After this, the instance is in ERROR state. If I delete it, it >> usually only release the IPs, and only after a while, if I re-run >> ``nova delete``, it *sometimes* actually delete the instance. >> >> This is driving me crazy, I don't really know where to look for >> hints. I guess the problem is in the network node, but I can't >> understand anything from the logs. >> >> can someone help me with this? >> >> Thank you in advance >> >> .a. >> >> -- >> [email protected] >> [email protected] +41 (0)44 635 42 22 >> GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ >> University of Zurich >> Winterthurerstrasse 190 >> CH-8057 Zurich Switzerland >> >> _______________________________________________ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : [email protected] >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > -- [email protected] [email protected] +41 (0)44 635 42 22 GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
