Hi Everybody,
I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for Cinder
volume storage and boot-from-volume, and I keep bumping into a very unhelpful
errors on my nova-compute test node and my cinder controller node.
Here is what I see on my cinder-volume controller (Node #1) when I try to
attach a RBD-backed Cinder volume to a Nova VM using either the GUI or nova
volume-attach (/var/log/cinder/volume.log):
2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp
[req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc
8431635821f84285afdd0f5faf1ce1aa] Exception during message handling
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line
441, in _process_data
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp **args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py",
line 148, in dispatch
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return
getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return
func(self, *args, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 624, in
initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
conn_info = self.driver.initialize_connection(volume, connector)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 665, in
initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp hosts,
ports = self._get_mon_addrs()
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 312, in
_get_mon_addrs
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp out, _
= self._execute(*args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return
processutils.execute(*cmd, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py",
line 158, in execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
shell=shell)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py", line 25, in
__init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib64/python2.6/subprocess.py", line 642, in __init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
errread, errwrite)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File
"/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp raise
child_exception
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp OSError:
[Errno 2] No such file or directory
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common
[req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc
8431635821f84285afdd0f5faf1ce1aa] Returning exception [Errno 2] No such file or
directory to caller
Here is what I see on my nova-compute node (Node #2) when I try to boot from
volume (/var/log/nova/compute.log):
ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa
9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has either erred or is
incapable of performing the requested operation. (HTTP 500) (Request-ID:
req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last):
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, in
_build_instance
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] context, instance, bdms)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in
_prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] instance=instance)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371, in
_prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] self._await_block_device_map_created)
+
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, in
attach_block_devices
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] block_device_mapping)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170, in
attach
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in wrapper
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] res = method(self, ctx, volume_id,
*args, **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in
initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306, in
initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] {'connector':
connector})[1]['connection_info']
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237, in
_action
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] return self.api.client.post(url,
body=body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] return self._cs_request(url, 'POST',
**kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in
_cs_request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] raise exceptions.from_response(resp,
body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has either
erred or is incapable of performing the requested operation. (HTTP 500)
(Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]
More info on my setup:
I'm running the most recent release of Dumpling libraries (0.67.4) and
python-ceph, but I do not have ceph itself installed as it should not be needed
on Havana
cephx is in use, and I set up pools and keyrings per the guide at
http://ceph.com/docs/next/rbd/rbd-openstack/
I'm using the ceph-extras repo to install the backported QEMU packages as
described at http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO
I'm also using Neutron+OVS and thus edited my qemu.conf according to this
libvirt wiki page
http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)
I am presently not configuring Nova to put its ephemeral disk image on RBD (new
option in Havana)
Things I've been able to do so far:
I stored an image to Glance backed by RBD
I used that image to create a Cinder volume backed by RBD
I instantiated a working ephemeral VM with Nova based on the image from Glance
backed by RBD
I created a libvirt VM manually with virsh on the same compute node that
attached a volume from the Cinder RBD pool
I created a VM with Nova (ephemeral local boot) that I was then able to use
virsh to successfully attach a Cinder volume backed by RBD
I can use qemu-img to create volumes in the cinder RBD pool, but only if I have
the client.admin keyring installed on the compute node
The error traced above is what happens every time I try to boot from volume
using that RBD-backed Cinder volume spawned from the RBD-backed Glance image.
What did work led me to believe that QEMU was the problem, so I tried the
following:
I changed the user and group qemu runs as from root to nova and from nova to
qemu to see if their permissions had any affect; no change
I tried the above tests with matching permissions on the contents of /etc/ceph/
(ceph.conf and the keyrings for admin, cinder, and glance); no change
It seems like Nova is somehow failing to get from Cinder that Cinder is using
RBD for its backend, but I can't understand why. All my configs align with
every instance of documentation I've been able to find for making OpenStack
work with Ceph. Has anyone done what I'm trying to do on CentOS 6 or even on
some version of Fedora? I am cross-posting this on the Ceph Users listserv as
well.
Thanks for your time,
Andy
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
