On Tue, Nov 19, 2019 at 1:51 PM shubjero <shubj...@gmail.com> wrote: > > Florian, > > Thanks for posting about this issue. This is something that we have > been experiencing (stale exclusive locks) with our OpenStack and Ceph > cloud more frequently as our datacentre has had some reliability > issues recently with power and cooling causing several unexpected > shutdowns. > > At this point we are on Ceph Mimic 13.2.6 and reading through this > thread and related links I just wanted to confirm if I have the > correct caps for cinder clients as listed below as we have upgraded > through many major Ceph versions over the years and I'm sure a lot of > our configs and settings still contain deprecated options. > > client.cinder > key: sanitized== > caps: [mgr] allow r > caps: [mon] profile rbd > caps: [osd] allow class-read object_prefix rbd_children, profile rbd > pool=volumes, profile rbd pool=vms, profile rbd pool=images
Only use "profile rbd" for 'mon' and 'osd' caps -- it's documented here [1]. Once you use 'profile rbd', you don't need the extra "allow class-read object_prefix rbd_children" since it is included within the profile (along with other things like support for clone v2). Octopus will also include "profile rbd" for the 'mgr' cap to support the new functionality in the "rbd_support" manager module (like running "rbd perf image top" w/o the admin caps). > From what I read, the blacklist permission was something that was > supposed to be applied pre-Luminous upgrade but once you are on > Luminous or later, it's no longer needed assuming you have switched to > using the rbd profile. Correct. The "blacklist" permission was an intermediate state pre-upgrade since your older OSDs wouldn't have support for "profile rbd" yet but Luminous OSDs started to enforce caps on the 'blacklist add' op so that rogue users w/ read-only permissions couldn't just blacklist all clients. Once you are at Luminous or later, you can just use the profile. > On Fri, Nov 15, 2019 at 11:05 AM Paul Emmerich <paul.emmer...@croit.io> wrote: > > > > To clear up a few misconceptions here: > > > > * RBD keyrings should use the "profile rbd" permissions, everything > > else is *wrong* and should be fixed asap > > * Manually adding the blacklist permission might work but isn't > > future-proof, fix the keyring instead > > * The suggestion to mount them elsewhere to fix this only works > > because "elsewhere" probably has an admin keyring, this is a bad > > work-around, fix the keyring instead > > * This is unrelated to openstack and will happen with *any* reasonably > > configured hypervisor that uses exclusive locking > > > > This problem usually happens after upgrading to Luminous without > > reading the change log. The change log tells you to adjust the keyring > > permissions accordingly > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io > > Tel: +49 89 1896585 90 > > > > On Fri, Nov 15, 2019 at 4:56 PM Joshua M. Boniface <jos...@boniface.me> > > wrote: > > > > > > Thanks Simon! I've implemented it, I guess I'll test it out next time my > > > homelab's power dies :-) > > > > > > On 2019-11-15 10:54 a.m., Simon Ironside wrote: > > > > > > On 15/11/2019 15:44, Joshua M. Boniface wrote: > > > > > > Hey All: > > > > > > I've also quite frequently experienced this sort of issue with my Ceph > > > RBD-backed QEMU/KVM > > > > > > cluster (not OpenStack specifically). Should this workaround of allowing > > > the 'osd blacklist' > > > > > > command in the caps help in that scenario as well, or is this an > > > OpenStack-specific > > > > > > functionality? > > > > > > Yes, my use case is RBD backed QEMU/KVM too, not Openstack. It's > > > required for all RBD clients. > > > > > > Simon > > > > > > _______________________________________________ > > > > > > ceph-users mailing list > > > > > > ceph-users@lists.ceph.com > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [1] https://docs.ceph.com/docs/master/rbd/rbd-openstack/#setup-ceph-client-authentication -- Jason _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
