Yesterday we tried to shut down all the services on the compute nodes and the controller node (including mysql and rabbitmq). For a little while everything seemed working fine, we were able to start a few instances very rapidly and without problems. After a while, however, the problem came up again:
* if the instance starts, it takes a lot of times (minutes) * 2 over 3 instances goes in error state * deleting instances takes *ages* Anyone willing to help me debugging this thing? .a. On Thu, Oct 31, 2013 at 6:28 PM, Antonio Messina <[email protected]> wrote: > Hi All, > > We are experiencing a problem with our OpenStack Folsom installation > on Ubuntu 12.04, using nova-network: instances are NOT created, and > deleting instances in error state fails. > > First of all: the installation was in "production" for almost one > year, and we didn't change anything. The only change made was updating > some packages from ubuntu and from ubuntu-cloud archives. > > (PS: The following text is formatted as restructured text) > > In details what happen is: > > * The instance is created `Thu Oct 31 18:01:00 CET 2013`:: > > 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi > [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] GET > http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d > 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi > [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] > http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d > returned with HTTP 200 > 2013-10-31 18:01:02 INFO nova.osapi_compute.wsgi.server > [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] 130.60.24.9 - - [31/Oct/2013 18:01:02] "GET > /v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d > HTTP/1.1" 200 1531 0.225107 > > * On the compute node, I see multiple times the following trace:: > > 2013-10-31 18:01:26 931 INFO nova.compute.resource_tracker [-] > Compute_service record updated for node-09-01-00 > 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free > ram (MB): 17389 > 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free > disk (GB): 136 > 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free > VCPUS: 0 > 2013-10-31 18:02:34 931 INFO nova.compute.resource_tracker [-] > Compute_service record updated for node-09-01-00 > 2013-10-31 18:02:41 931 ERROR nova.openstack.common.rpc.amqp [-] > Exception during message handling > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp Traceback > (most recent call last): > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 276, in _process_data > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp rval = > self.proxy.dispatch(ctxt, version, method, **args) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", > line 145, in dispatch > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return > getattr(proxyobj, method)(ctxt, **kwargs) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp > temp_level, payload) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return > f(*args, **kw) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 390, in > refresh_instance_security_rules > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return > self.driver.refresh_instance_security_rules(instance) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2243, in > refresh_instance_security_rules > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp > self.firewall_driver.refresh_instance_security_rules(instance) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 446, in > refresh_instance_security_rules > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp > self.do_refresh_instance_rules(instance) > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 463, in > do_refresh_instance_rules > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp > network_info = self.network_infos[instance['id']] > 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp KeyError: > 38331 > > This backtrace does not seem to be relevant, as I've already seen > it in log of instances that started without problems. > > * After a while, IPs are assigned to the instance. I don't see > anything in the ``nova-network.log``, but I can see the IPs in the > ``nova list`` output. I also see the following errors in the compute > node:: > > 2013-10-31 18:04:04 931 ERROR nova.openstack.common.rpc.common [-] > Timed out waiting for RPC response: timed out > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common > Traceback (most recent call last): > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 518, in ensure > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > method(*args, **kwargs) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 596, in _consume > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > self.connection.drain_events(timeout=timeout) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in > drain_events > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > self.transport.drain_events(self.connection, **kwargs) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in > drain_events > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > connection.drain_events(**kwargs) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in > drain_events > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > self.wait_multi(self.channels.values(), timeout=timeout) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in > wait_multi > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common > chanmap.keys(), allowed_methods, timeout=timeout) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in > _wait_multiple > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common > channel, method_sig, args, content = read_timeout(timeout) > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in > read_timeout > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return > self.method_reader.read_method() > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File > "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line > 221, in read_method > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common > raise m > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common timeout: > timed out > 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common > 2013-10-31 18:04:04 ERROR nova.compute.manager > [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] Instance failed network setup > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] Traceback (most recent call last): > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in > _allocate_network > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] requested_networks=requested_networks) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] res = f(self, context, *args, > **kwargs) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in > allocate_for_instance > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] 'args': args}) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", > line 108, in call > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] return _get_impl().call(cfg.CONF, > context, topic, msg, timeout) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 718, in call > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] rpc_amqp.get_connection_pool(conf, > Connection)) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 369, in call > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] rv = list(rv) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 330, in __iter__ > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] self.done() > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] self.gen.next() > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 327, in __iter__ > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] self._iterator.next() > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 601, in iterconsume > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] yield self.ensure(_error_callback, > _consume) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 521, in ensure > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] error_callback(e) > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 582, in _error_callback > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] raise rpc_common.Timeout() > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] Timeout: Timeout while waiting on RPC > response. > 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] > 2013-10-31 18:04:04 INFO nova.compute.resource_tracker > [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] Aborting claim: [Claim > ece6db13-10ee-4808-9646-57ab285ca33d: 14336 MB memory, 100 GB disk, 8 VCPUS] > > and then again:: > > 2013-10-31 18:07:04 ERROR nova.compute.manager > [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 > 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: > ece6db13-10ee-4808-9646-57ab285ca33d] Build error: ['Traceback (most recent > call last):\n', ' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 498, in > _run_instance\n requested_networks)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in > _allocate_network\n requested_networks=requested_networks)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper\n > res = f(self, context, *args, **kwargs)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in > allocate_for_instance\n \'args\': args})\n', ' File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", > line 108, in call\n return _get_impl().call(cfg.CONF, context, topic, msg, > timeout)\n', ' File "/usr/lib/python2.7/di st-packages/nova/openstack/common/rpc/impl_kombu.py", line 718, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369, in call\n rv = list(rv)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__\n self.done()\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__\n self._iterator.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 601, in iterconsume\n yield self.ensure(_error_callback, _consume)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 521, in ensure\n error_callback(e)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 582, in _error_callback\n raise rpc_common.Timeout()\n', 'Timeout: Timeout while waiting on RPC response.\n'] > 2013-10-31 18:07:05 931 ERROR nova.openstack.common.rpc.amqp [-] > Exception during message handling > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Traceback > (most recent call last): > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 276, in _process_data > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rval = > self.proxy.dispatch(ctxt, version, method, **args) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", > line 145, in dispatch > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return > getattr(proxyobj, method)(ctxt, **kwargs) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > temp_level, payload) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return > f(*args, **kw) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in > decorated_function > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp pass > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in > decorated_function > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return > function(self, context, *args, **kwargs) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in > decorated_function > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > kwargs['instance']['uuid'], e, sys.exc_info()) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in > decorated_function > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return > function(self, context, *args, **kwargs) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in > run_instance > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > do_run_instance() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp retval = > f(*args, **kwargs) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in > do_run_instance > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > admin_password, is_first_time, instance) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in > _run_instance > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self._set_instance_error_state(context, instance['uuid']) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in > _run_instance > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > is_first_time, request_spec, filter_properties) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 547, in > _reschedule_or_reraise > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self._deallocate_network(context, instance) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 783, in > _deallocate_network > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.network_api.deallocate_for_instance(context, instance) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 261, in > deallocate_for_instance > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp 'args': > args}) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", > line 108, in call > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return > _get_impl().call(cfg.CONF, context, topic, msg, timeout) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 718, in call > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > rpc_amqp.get_connection_pool(conf, Connection)) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 369, in call > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rv = > list(rv) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 330, in __iter__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.done() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self.gen.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 327, in __iter__ > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > self._iterator.next() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 601, in iterconsume > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp yield > self.ensure(_error_callback, _consume) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 521, in ensure > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > error_callback(e) > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 582, in _error_callback > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp raise > rpc_common.Timeout() > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Timeout: > Timeout while waiting on RPC response. > 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp > > * After this, the instance is in ERROR state. If I delete it, it > usually only release the IPs, and only after a while, if I re-run > ``nova delete``, it *sometimes* actually delete the instance. > > This is driving me crazy, I don't really know where to look for > hints. I guess the problem is in the network node, but I can't > understand anything from the logs. > > can someone help me with this? > > Thank you in advance > > .a. > > -- > [email protected] > [email protected] +41 (0)44 635 42 22 > GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ > University of Zurich > Winterthurerstrasse 190 > CH-8057 Zurich Switzerland -- [email protected] [email protected] +41 (0)44 635 42 22 GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
