Hi Vasilly, Thanks, but I still see the same error: cinder.conf (of course I just restart the cinder-volume service)
# default volume type to use (string value) [rbd-cephvolume] rbd_user = cinder rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx volume_backend_name=rbd volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_pool = volumes rbd_ceph_conf = /etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 glance_api_version = 2 xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory xen be: qdisk-51760: initialise() failed Regards, I 2015-11-06 13:00 GMT+01:00 Vasiliy Angapov <anga...@gmail.com>: > At cinder.conf you should place this options: > > rbd_user = cinder > rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx > > to [rbd-cephvolume] section instead of DEFAULT. > > 2015-11-06 19:45 GMT+08:00 Iban Cabrillo <cabri...@ifca.unican.es>: > > Hi, > > One more step debugging this issue (hypervisor/nova-compute node is XEN > > 4.4.2): > > > > I think the problem is that libvirt is not getting the correct user or > > credentials tu access pool, on instance qemu log i see: > > > > xen be: qdisk-51760: error: Could not open > > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > > directory > > xen be: qdisk-51760: initialise() failed > > xen be: qdisk-51760: error: Could not open > > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > > directory > > xen be: qdisk-51760: initialise() failed > > xen be: qdisk-51760: error: Could not open > > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > > directory > > > > But using the user cinder on pool volumes : > > > > rbd ls -p volumes --id cinder > > test > > volume-4d26bb31-91e8-4646-8010-82127b775c8e > > volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a > > volume-7da08f12-fb0f-4269-931a-d528c1507fee > > > > Using: > > qemu-img info -f rbd rbd:volumes/test > > Does not work, but using directly the user cinder and the ceph.conf file > > works fine: > > > > qemu-img info -f rbd rbd:volumes/test:id=cinder:conf=/etc/ceph/ceph.conf > > > > I think nova.conf is set correctly (section libvirt): > > images_rbd_pool = volumes > > images_rbd_ceph_conf = /etc/ceph/ceph.conf > > hw_disk_discard=unmap > > rbd_user = cinder > > rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX > > > > And looking at libvirt: > > > > # virsh secret-list > > setlocale: No such file or directory > > UUID Usage > > > -------------------------------------------------------------------------------- > > 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX ceph client.cinder secret > > > > > > virsh secret-get-value 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX > > setlocale: No such file or directory > > AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg== > > cat /etc/ceph/ceph.client.cinder.keyring > > [client.cinder] > > key = AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg== > > > > > > Any idea will be welcomed. > > regards, I > > > > 2015-11-04 10:51 GMT+01:00 Iban Cabrillo <cabri...@ifca.unican.es>: > >> > >> Dear Cephers, > >> > >> I still can attach volume to my cloud machines, ceph version is > 0.94.5 > >> (9764da52395923e0b32908d83a9f7304401fee43) and Openstack Juno > >> > >> Nova+cinder are able to create volumes on Ceph > >> cephvolume:~ # rados ls --pool volumes > >> rbd_header.1f7784a9e1c2e > >> rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a > >> rbd_directory > >> rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee > >> rbd_header.23d5e33b4c15c > >> rbd_id.volume-4d26bb31-91e8-4646-8010-82127b775c8e > >> rbd_header.20407190ce77f > >> > >> cloud:~ # cinder list > >> > >> > +--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+ > >> | ID | > >> Status | Display Name | Size | Volume Type | Bootable | > >> Attached to | > >> > >> > +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------|---------------------------------------------------------+ > >> | 4d26bb31-91e8-4646-8010-82127b775c8e | in-use | None | > 2 > >> | rbd | false | > 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb > >> | > >> > >> > +--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+ > >> > >> > >> nova:~ # nova volume-attach 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb > >> 4d26bb31-91e8-4646-8010-82127b775c8e auto > >> > +----------+------------------------------------------------------------+ > >> | Property | Value > >> | > >> > +----------+------------------------------------------------------------+ > >> | device | /dev/xvdd > >> | > >> | id | 4d26bb31-91e8-4646-8010-82127b775c8e | > >> | serverId | 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb | > >> | volumeId | 4d26bb31-91e8-4646-8010-82127b775c8e | > >> +----------+--------------------------------------+ > >> > >> From nova-compute (Ubuntu 14.04 LTS \n \l) node I see the > >> attaching/detaching: > >> cloud01:~ # dpkg -l | grep ceph > >> ii ceph-common 0.94.5-1trusty > >> amd64 common utilities to mount and interact with a ceph storage > >> cluster > >> ii libcephfs1 0.94.5-1trusty > >> amd64 Ceph distributed file system client library > >> ii python-cephfs 0.94.5-1trusty > >> amd64 Python libraries for the Ceph libcephfs library > >> ii librbd1 0.94.5-1trusty > >> amd64 RADOS block device client library > >> ii python-rbd 0.94.5-1trusty > >> amd64 Python libraries for the Ceph librbd library > >> > >> at cinder.conf > >> > >> rbd_user = cinder > >> rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx > >> > >> [rbd-cephvolume] > >> volume_backend_name=rbd > >> volume_driver = cinder.volume.drivers.rbd.RBDDriver > >> rbd_pool = volumes > >> rbd_ceph_conf = /etc/ceph/ceph.conf > >> rbd_flatten_volume_from_snapshot = false > >> rbd_max_clone_depth = 5 > >> rbd_store_chunk_size = 4 > >> rados_connect_timeout = -1 > >> glance_api_version = 2 > >> > >> in nova.conf > >> rbd_user=cinder > >> > >> # The libvirt UUID of the secret for the rbd_uservolumes > >> # (string value) > >> rbd_secret_uuid=67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx > >> > >> images_rbd_pool=volumes > >> > >> # Path to the ceph configuration file to use (string value) > >> images_rbd_ceph_conf=/etc/ceph/ceph.conf > >> > >> ls -la /etc/libvirt/secrets > >> total 16 > >> drwx------ 2 root root 4096 Nov 4 10:28 . > >> drwxr-xr-x 7 root root 4096 Oct 22 13:15 .. > >> -rw------- 1 root root 40 Nov 4 10:28 > >> 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxx.base64 > >> -rw------- 1 root root 170 Nov 4 10:25 > >> 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxx.xml > >> > >> > >> > >> 2015-11-04 10:39:42.573 11653 INFO nova.compute.manager > >> [req-8b2a9793-4b39-4cb0-b291-e492c350387e > b7aadbb4a85745feb498b74e437129cc > >> ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: > >> 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Detach volume > >> 4d26bb31-91e8-4646-8010-82127b775c8e from mountpoint /dev/xvdd > >> 2015-11-04 10:40:43.266 11653 INFO nova.compute.manager > >> [req-35218de0-3f26-496b-aad9-5c839143da17 > b7aadbb4a85745feb498b74e437129cc > >> ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: > >> 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Attaching volume > >> 4d26bb31-91e8-4646-8010-82127b775c8e to /dev/xvdd > >> > >> but one on cloud machine (SL6) the volume y never showed (xvdd). > >> [root@cloud5 ~]# cat /proc/partitions > >> major minor #blocks name > >> > >> 202 0 20971520 xvda > >> 202 16 209715200 xvdb > >> 202 32 10485760 xvdc > >> > >> Thanks in advance, I > >> > >> 2015-11-03 11:18 GMT+01:00 Iban Cabrillo <cabri...@ifca.unican.es>: > >>> > >>> Hi all, > >>> During last week I been trying to deploy the pre-existing ceph > >>> cluster with out openstack intance. > >>> The ceph-cinder integration was easy (or at least I think so!!) > >>> There is only one volume to attach block storage to out cloud > >>> machines. > >>> > >>> The client.cinder has permission on this volume (following the > >>> guides) > >>> ............... > >>> client.cinder > >>> key: AQAonXXXXXXXRAAPIAj9iErv001a0k+vyFdUg== > >>> caps: [mon] allow r > >>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx > >>> pool=volumes > >>> > >>> ceph.conf file seems to be OK: > >>> > >>> [global] > >>> fsid = 6f5a65a7-316c-4825-afcb-428608941dd1 > >>> mon_initial_members = cephadm, cephmon02, cephmon03 > >>> mon_host = 10.10.3.1,10.10.3.2,10.10.3.3 > >>> auth_cluster_required = cephx > >>> auth_service_required = cephx > >>> auth_client_required = cephx > >>> filestore_xattr_use_omap = true > >>> osd_pool_default_size = 2 > >>> public_network = 10.10.0.0/16 > >>> cluster_network = 192.168.254.0/27 > >>> > >>> [osd] > >>> osd_journal_size = 20000 > >>> > >>> [client.cinder] > >>> keyring = /etc/ceph/ceph.client.cinder.keyring > >>> > >>> [client] > >>> rbd cache = true > >>> rbd cache writethrough until flush = true > >>> admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok > >>> > >>> > >>> The trouble seems that blocks are created using the client.admin > instead > >>> of client.cinder > >>> > >>> From cinder machine: > >>> > >>> cinder:~ # rados ls --pool volumes > >>> rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a > >>> rbd_directory > >>> rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee > >>> rbd_header.23d5e33b4c15c > >>> rbd_header.20407190ce77f > >>> > >>> But if I try to look for using cinder client: > >>> > >>> > >>> cinder:~ #rados ls --pool volumes --secret client.cinder > >>> "empty answer" > >>> > >>> cinder:~ # ls -la /etc/ceph > >>> total 24 > >>> drwxr-xr-x 2 root root 4096 nov 3 10:17 . > >>> drwxr-xr-x 108 root root 4096 oct 29 09:52 .. > >>> -rw------- 1 root root 63 nov 3 10:17 > ceph.client.admin.keyring > >>> -rw-r--r-- 1 cinder cinder 67 oct 28 13:44 > ceph.client.cinder.keyring > >>> -rw-r--r-- 1 root root 454 oct 1 13:56 ceph.conf > >>> -rw-r--r-- 1 root root 73 sep 27 09:36 ceph.mon.keyring > >>> > >>> > >>> from a client (I have supposed that this machine only need the cinder > >>> key...) > >>> > >>> cloud28:~ # ls -la /etc/ceph/ > >>> total 28 > >>> drwx------ 2 root root 4096 nov 3 11:01 . > >>> drwxr-xr-x 116 root root 12288 oct 30 14:37 .. > >>> -rw-r--r-- 1 nova nova 67 oct 28 11:43 ceph.client.cinder.keyring > >>> -rw-r--r-- 1 root root 588 nov 3 10:59 ceph.conf > >>> -rw-r--r-- 1 root root 92 oct 26 16:59 rbdmap > >>> > >>> cloud28:~ # rbd -p volumes ls > >>> 2015-11-03 11:01:58.782795 7fc6c714b840 -1 monclient(hunting): ERROR: > >>> missing keyring, cannot use cephx for authentication > >>> 2015-11-03 11:01:58.782800 7fc6c714b840 0 librados: client.admin > >>> initialization error (2) No such file or directory > >>> rbd: couldn't connect to the cluster! > >>> > >>> Any help will be welcome. > >>> > >> > >> > >> > >> -- > >> > >> > ############################################################################ > >> Iban Cabrillo Bartolome > >> Instituto de Fisica de Cantabria (IFCA) > >> Santander, Spain > >> Tel: +34942200969 > >> PGP PUBLIC KEY: > >> http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC > >> > >> > ############################################################################ > >> Bertrand Russell: > >> "El problema con el mundo es que los estúpidos están seguros de todo y > los > >> inteligentes están llenos de dudas" > > > > > > > > > > -- > > > ############################################################################ > > Iban Cabrillo Bartolome > > Instituto de Fisica de Cantabria (IFCA) > > Santander, Spain > > Tel: +34942200969 > > PGP PUBLIC KEY: > > http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC > > > ############################################################################ > > Bertrand Russell: > > "El problema con el mundo es que los estúpidos están seguros de todo y > los > > inteligentes están llenos de dudas" > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- ############################################################################ Iban Cabrillo Bartolome Instituto de Fisica de Cantabria (IFCA) Santander, Spain Tel: +34942200969 PGP PUBLIC KEY: http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC ############################################################################ Bertrand Russell: *"El problema con el mundo es que los estúpidos están seguros de todo y los inteligentes están llenos de dudas*"
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
