Hi all,
I am having a terrible time getting my instances to work after a hard reboot. I am using the most up-to date version of all openstack packages provided by Ubuntu. I have included a list of packages, with version, at the end of this email. After a hard reboot "nova list" reports that the instance is active, but there are no kvm processes running. grepping the log file for errors I find this in nova-compute.log: 2012-08-09 14:32:51 INFO nova.rpc.common [req-dd6fcade-73ec-4378-9a6b-7bc709eefcd4 None None] Connected to AMQP server on cloudy-priv:5672 2012-08-09 14:33:51 ERROR nova.rpc.common [req-dd6fcade-73ec-4378-9a6b-7bc709eefcd4 None None] Timed out waiting for RPC response: timed out 2012-08-09 14:33:51 TRACE nova.rpc.common Traceback (most recent call last): 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure 2012-08-09 14:33:51 TRACE nova.rpc.common return method(*args, **kwargs) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume 2012-08-09 14:33:51 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events 2012-08-09 14:33:51 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events 2012-08-09 14:33:51 TRACE nova.rpc.common return connection.drain_events(**kwargs) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events 2012-08-09 14:33:51 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi 2012-08-09 14:33:51 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple 2012-08-09 14:33:51 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout 2012-08-09 14:33:51 TRACE nova.rpc.common return self.method_reader.read_method() 2012-08-09 14:33:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method 2012-08-09 14:33:51 TRACE nova.rpc.common raise m 2012-08-09 14:33:51 TRACE nova.rpc.common timeout: timed out 2012-08-09 14:33:51 TRACE nova.rpc.common 2012-08-09 14:33:51 CRITICAL nova [-] Timeout while waiting on RPC response. restarting nova-compute brings the instance up, so it looks like nova-compute is starting before rabbitmq? Is there a clean way around this, or should I put "service nova-compute restart" in rc.local? If I have a volume attached things get much worse. I can still start the instance by restarting nova-compute, but the volume does not attach. I can not seem to detach the volume in order to attach it again. Below is the only error in the log file, and how I mount the image that contains the nova-volume logical group. The error occurs because it tries to start nova-volume before the loopback device is setup. The command in rc.local restarts the service, making the logical group available. >From nova-volume.log 2012-08-09 14:32:40 CRITICAL nova [-] volume group nova-volumes doesn't exist >From rc.local losetup -f /var/lib/nova/nova-volumes.img service nova-volume restart Any idea how I should solve these problems? I could disable upstart from bringing the services up automatically and start them in the correct order in rc.local, but I don't think this would solve the volume attachment issue. I am so frustrated that I created this script for testing which completely resets the nova database table, iptables, and recreates everything. http://paste2.org/p/2100211 I know it is a dirty dirty hack, but I can't seem to figure out what is going on. Thanks in advance for the help. Sam root@cloudy:/var/log/nova# dpkg -l | grep -E "(nova|glance|keystone|tgt|rabbit|ntp|mysql|libvirt|kvm)" ii glance 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - Daemons ii glance-api 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - API ii glance-client 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - Registry ii glance-common 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - Common ii glance-registry 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - Registry ii keystone 2012.1+stable~20120608-aff45d6-0ubuntu1 OpenStack identity service - Daemons ii kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.1 dummy transitional package from kvm to qemu-kvm ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 PXE ROM's for KVM ii libdbd-mysql-perl 4.020-1build2 Perl5 database interface to the MySQL database ii libmysqlclient18 5.5.24-0ubuntu0.12.04.1 MySQL database client library ii libsys-virt-perl 0.9.7-2 Perl module providing an extension for the libvirt library ii libvirt-bin 0.9.8-2ubuntu17.3 programs for the libvirt library ii libvirt0 0.9.8-2ubuntu17.3 library for interfacing with different virtualization systems ii mysql-client-5.5 5.5.24-0ubuntu0.12.04.1 MySQL database client binaries ii mysql-client-core-5.5 5.5.24-0ubuntu0.12.04.1 MySQL database core client binaries ii mysql-common 5.5.24-0ubuntu0.12.04.1 MySQL database common files, e.g. /etc/mysql/my.cnf ii mysql-server 5.5.24-0ubuntu0.12.04.1 MySQL database server (metapackage depending on the latest version) ii mysql-server-5.5 5.5.24-0ubuntu0.12.04.1 MySQL database server binaries and system database setup ii mysql-server-core-5.5 5.5.24-0ubuntu0.12.04.1 MySQL database server binaries ii nova-api 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - API frontend ii nova-common 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - common files ii nova-compute 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - compute node ii nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - compute node (KVM) ii nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - Network manager ii nova-scheduler 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - virtual machine scheduler ii nova-volume 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute - storage ii ntp 1:4.2.6.p3+dfsg-1ubuntu3.1 Network Time Protocol daemon and utility programs ii ntpdate 1:4.2.6.p3+dfsg-1ubuntu3.1 client for setting system time from NTP servers ii python-glance 2012.1+stable~20120608-5462295-0ubuntu2.2 OpenStack Image Registry and Delivery Service - Python library ii python-keystone 2012.1+stable~20120608-aff45d6-0ubuntu1 OpenStack identity service - Python library ii python-keystoneclient 2012.1-0ubuntu1 Client libary for Openstack Keystone API ii python-libvirt 0.9.8-2ubuntu17.3 libvirt Python bindings ii python-mysqldb 1.2.3-1build1 Python interface to MySQL ii python-nova 2012.1+stable~20120612-3ee026e-0ubuntu1.2 OpenStack Compute Python libraries ii python-novaclient 2012.1-0ubuntu1 client library for OpenStack Compute API ii qemu-kvm 1.0+noroms-0ubuntu14.1 Full virtualization on i386 and amd64 hardware ii rabbitmq-server 2.7.1-0ubuntu4 An AMQP server written in Erlang ii tgt 1:1.0.17-1ubuntu2 Linux SCSI target user-space tools _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
