Hi Mike, That led me to realise what the issue was. My cinder (volumes) client did not have the correct perms on the images pool. I ran the following to update the perms for that client:
ceph auth caps client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' ...and was then able to successfully boot an instance from a cinder volume that was created by cloning a glance image from the images pool! One last question: I presume the fact that the 'volume_image_metadata' field is not populated when cloning a glance image into a cinder volume is a bug? It means that the cinder client doesn't show the volume as bootable, though I'm not sure what other detrimental effect it actually has (clearly the volume can be booted from). Thanks Darren On 10 September 2013 21:04, Darren Birkett <[email protected]> wrote: > Hi Mike, > > Thanks - glad to hear it definitely works as expected! Here's my > client.glance and client.volumes from 'ceph auth list': > > client.glance > key: AQAWFi9SOKzAABAAPV1ZrpWkx72tmJ5E7nOi3A== > caps: [mon] allow r > caps: [osd] allow rwx pool=images, allow class-read object_prefix > rbd_children > client.volumes > key: AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A== > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx > pool=volumes > > Thanks > Darren > > > On 10 September 2013 20:08, Mike Dawson <[email protected]> wrote: > >> Darren, >> >> I can confirm Copy on Write (show_image_direct_url = True) does work in >> Grizzly. >> >> It sounds like you are close. To check permissions, run 'ceph auth list', >> and reply with "client.images" and "client.volumes" (or whatever keys you >> use in Glance and Cinder). >> >> Cheers, >> Mike Dawson >> >> >> >> On 9/10/2013 10:12 AM, Darren Birkett wrote: >> >>> Hi All, >>> >>> tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly? >>> >>> I'm currently testing a ceph/rados back end with an openstack >>> installation. I have the following things working OK: >>> >>> 1. cinder configured to create volumes in RBD >>> 2. nova configured to boot from RBD backed cinder volumes (libvirt UUID >>> secret set etc) >>> 3. glance configured to use RBD as a back end store for images >>> >>> With this setup, when I create a bootable volume in cinder, passing an >>> id of an image in glance, the image gets downloaded, converted to raw, >>> and then created as an RBD object and made available to cinder. The >>> correct metadata field for the cinder volume is populated >>> (volume_image_metadata) and so the cinder client marks the volume as >>> bootable. This is all fine. >>> >>> If I want to take advantage of the fact that both glance images and >>> cinder volumes are stored in RBD, I can add the following flag to the >>> glance-api.conf: >>> >>> show_image_direct_url = True >>> >>> This enables cinder to see that the glance image is stored in RBD, and >>> the cinder rbd driver then, instead of downloading the image and >>> creating an RBD image from it, just issues an 'rbd clone' command (seen >>> in the cinder-volume.log): >>> >>> rbd clone --pool images --image dcb2f16d-a09d-4064-9198-**1965274e214d >>> --snap snap --dest-pool volumes --dest >>> volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d >>> >>> This is all very nice, and the cinder volume is available immediately as >>> you'd expect. The problem is that the metadata field is not populated >>> so it's not seen as bootable. Even manually populating this field >>> leaves the volume unbootable. The volume can not even be attached to >>> another instance for inspection. >>> >>> libvirt doesn't seem to be able to access the rbd device. From >>> nova-compute.log: >>> >>> qemu-system-x86_64: -drive >>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-** >>> fa667bd47c6d:id=volumes:key=**AQAnAy9ScPB4IRAAtxD/** >>> V1rDciqFiT9AMPPr+A==:auth_**supported=cephx\;none,if=none,** >>> id=drive-virtio-disk0,format=**raw,serial=20987f9d-b4fb-463d-** >>> 8b8f-fa667bd47c6d,cache=none: >>> error reading header from volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d >>> >>> qemu-system-x86_64: -drive >>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-** >>> fa667bd47c6d:id=volumes:key=**AQAnAy9ScPB4IRAAtxD/** >>> V1rDciqFiT9AMPPr+A==:auth_**supported=cephx\;none,if=none,** >>> id=drive-virtio-disk0,format=**raw,serial=20987f9d-b4fb-463d-** >>> 8b8f-fa667bd47c6d,cache=none: >>> could not open disk image >>> rbd:volumes/volume-20987f9d-**b4fb-463d-8b8f-fa667bd47c6d:** >>> id=volumes:key=**AQAnAy9ScPB4IRAAtxD/**V1rDciqFiT9AMPPr+A==:auth_** >>> supported=cephx\;none: >>> Operation not permitted >>> >>> It's almost like a permission issue, but my ceph/rbd knowledge is still >>> fledgeling. >>> >>> I know that the cinder rbd driver has been rewritten to use librbd in >>> havana, and I'm wondering if this will change any of this behaviour? >>> I'm also wondering if anyone has actually got this working with >>> grizzly, and how? >>> >>> Many thanks >>> Darren >>> >>> >>> >>> ______________________________**_________________ >>> ceph-users mailing list >>> [email protected] >>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>> >>> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
