Thanks for the response Josh! If the Ceph CLI tool still needs to be there for Cinder in Havana, then am I correct in assuming that I still also need to export "CEPH_ARGS='--id volumes'" in my cinder init script for the sake of cephx like I had to do in Grizzly?
Thanks, Andy On Oct 21, 2013, at 12:26 PM, Josh Durgin <[email protected]> wrote: > On 10/21/2013 09:03 AM, Andrew Richards wrote: >> Hi Everybody, >> >> I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for >> Cinder volume storage and boot-from-volume, and I keep bumping into a >> very unhelpful errors on my nova-compute test node and my cinder >> controller node. >> >> Here is what I see on my cinder-volume controller (Node #1) when I try >> to attach a RBD-backed Cinder volume to a Nova VM using either the GUI >> or nova volume-attach (/var/log/cinder/volume.log): >> >> 2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp >> [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 >> 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] >> Exception during message handling >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> Traceback (most recent call last): >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File >> "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", >> line 441, in _process_data >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> **args) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File >> "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", >> line 148, in dispatch >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> return getattr(proxyobj, method)(ctxt, **kwargs) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in >> wrapper >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> return func(self, *args, **kwargs) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line >> 624, in initialize_connection >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> conn_info = self.driver.initialize_connection(volume, connector) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", >> line 665, in initialize_connection >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> hosts, ports = self._get_mon_addrs() >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", >> line 312, in _get_mon_addrs >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> out, _ = self._execute(*args) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in >> execute >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> return processutils.execute(*cmd, **kwargs) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File >> "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", >> line 158, in execute >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> shell=shell) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py", >> line 25, in __init__ >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__ >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> errread, errwrite) >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> raise child_exception >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> OSError: [Errno 2] No such file or directory >> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp >> 2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common >> [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 >> 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] >> Returning exception [Errno 2] No such file or directory to caller >> >> >> Here is what I see on my nova-compute node (Node #2) when I try to boot >> from volume (/var/log/nova/compute.log): >> >> ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa >> 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] >> [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has >> either erred or is incapable of performing the requested operation. >> (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last): >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, >> in _build_instance >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] context, instance, bdms) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, >> in _prep_block_device >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] instance=instance) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371, >> in _prep_block_device >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] >> self._await_block_device_map_created) + >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, >> in attach_block_devices >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] block_device_mapping) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170, >> in attach >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] connector) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in >> wrapper >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] res = method(self, ctx, >> volume_id, *args, **kwargs) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in >> initialize_connection >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] connector) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306, >> in initialize_connection >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] {'connector': >> connector})[1]['connection_info'] >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237, >> in _action >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] return >> self.api.client.post(url, body=body) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] return self._cs_request(url, >> 'POST', **kwargs) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in >> _cs_request >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] **kwargs) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] File >> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in >> request >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] raise >> exceptions.from_response(resp, body) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has >> either erred or is incapable of performing the requested operation. >> (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b) >> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: >> c80a053f-b84c-401c-8e29-022d4c6f56a0] >> >> >> More info on my setup: >> >> * I'm running the most recent release of Dumpling libraries (0.67.4) >> and python-ceph, but I do not have ceph itself installed as it >> should not be needed on Havana > > This is where the error is coming from - although most of the rbd > driver was converted to use librbd, there are a couple things that > still rely on the cli tools, since they had some functionality that > wasn't present in librados in cuttlefish or bobtail. > > Specifically, the attach is failing because cinder is trying to use the > 'ceph' command to get a list of all monitors to pass to nova. Installing > ceph-common on the node running cinder-volume should fix it. > >> * cephx is in use, and I set up pools and keyrings per the guide at >> http://ceph.com/docs/next/rbd/rbd-openstack/ > > These aren't updated for havana yet, but hopefully your email helps in > the interim. This is the main new requirement - librbd python bindings > and cli tools are needed for cinder-volume. > > Josh > >> * I'm using the ceph-extras repo to install the backported QEMU >> packages as described at >> http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO >> * I'm also using Neutron+OVS and thus edited my qemu.conf according to >> this libvirt wiki page >> >> http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface) >> >> <http://wiki.libvirt.org/page/Guest_won%27t_start_-_warning:_could_not_open_/dev/net/tun_%28%27generic_ethernet%27_interface%29> >> * I am presently not configuring Nova to put its ephemeral disk image >> on RBD (new option in Havana) >> >> >> Things I've been able to do so far: >> >> * I stored an image to Glance backed by RBD >> * I used that image to create a Cinder volume backed by RBD >> * I instantiated a working ephemeral VM with Nova based on the image >> from Glance backed by RBD >> * I created a libvirt VM manually with virsh on the same compute node >> that attached a volume from the Cinder RBD pool >> * I created a VM with Nova (ephemeral local boot) that I was then able >> to use virsh to successfully attach a Cinder volume backed by RBD >> * I can use qemu-img to create volumes in the cinder RBD pool, but >> only if I have the client.admin keyring installed on the compute node >> >> >> The error traced above is what happens every time I try to boot from >> volume using that RBD-backed Cinder volume spawned from the RBD-backed >> Glance image. What did work led me to believe that QEMU was the >> problem, so I tried the following: >> >> * I changed the user and group qemu runs as from root to nova and from >> nova to qemu to see if their permissions had any affect; no change >> * I tried the above tests with matching permissions on the contents of >> /etc/ceph/ (ceph.conf and the keyrings for admin, cinder, and >> glance); no change >> >> >> It seems like Nova is somehow failing to get from Cinder that Cinder is >> using RBD for its backend, but I can't understand why. All my configs >> align with every instance of documentation I've been able to find for >> making OpenStack work with Ceph. Has anyone done what I'm trying to do >> on CentOS 6 or even on some version of Fedora? I am cross-posting this >> on the OpenStack listserv as well. >> >> Thanks for your time, >> >> Andy >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
