On 10/07/18 04:40, Gregory Farnum wrote: > On Sun, Jul 8, 2018 at 6:06 PM Graeme Gillies <[email protected] > <mailto:[email protected]>> wrote: > > Hi, > > I was wondering how (if?) people handle rotating cephx keys while > keeping cluster up/available. > > Part of meeting compliance standards such as PCI DSS is making > sure that > data encryption keys and security credentials are rotated > regularly and > during other key points (such as notable staff turnover). > > We are currently looking at using Ceph as a storage solution and was > wondering how people handle rotating cephx keys (at the very > least, the > admin and client.$user keys) while causing minimal/no downtime to ceph > or the clients. > > My understanding is that if you change the keys stored in the ceph > kv db > then any existing sessions should still continue to work, but any new > ones (say, a hypervisor establishing new connections to osds for a new > vm volume) will fail until the key on the client side is also updated. > > I attempted to set two keys against the same client to see if I > can have > an "overlap" period of new and old keys before rotating out the > old key, > but it seems that ceph only has the concept of 1 key per user. > > Any hints, advice, or any information on how to achieve this would be > much appreciated. > > > This isn't something I've seen come up much. Your understanding sounds > correct to me, so as a naive developer I'd assume you just change the > key in the monitors and distribute the new one to whoever should have > it. There's a small window in which the admin with the old key can't > do anything, but presumably you can coordinate around that? > > The big issue I'm aware of is that orchestration systems like > OpenStack don't always do a good job supporting those changes — eg, I > think it embeds some keys in its database descriptor for the rbd > volume? :/ > -Greg
I think the biggest problem with simply changing keys is that lets say I have a client connecting to ceph using a ceph.client.user account. If I want to rotate the keys for that I can simply do that ceph cluster side, but then I also need to do that on the client side (in my case virtual machine hypervisors). DUring this window (which might be tiny with decent tooling, but still non-zero) my clients can't do new connections to the ceph cluster, which I assume will cause issues. I do wonder if an RFE to allow ceph auth to accept multiple keys for client would be accepted? That way I would add my new key to the ceph auth (so clients can authenticate with either key), then rotate it out on my hypervisors, then remove the old key from ceph auth when done. As for Openstack, when I used it I was pretty sure it simply used ceph.conf of the nova-compute hosts to connect to ceph (at least for libvirt) however that doesn't mean it does something else for other hypervisors or implementations. Regards, Graeme > > > > Thanks in advance, > > Graeme > > _______________________________________________ > ceph-users mailing list > [email protected] <mailto:[email protected]> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=xj3ftZ-YqnF9lrwuKqTuKqnVjnR0xteMhkRGTrOYYdo&m=zjBoDFxwgvY8z1hrXjQlTuTT-ahxsiJPmUGoLYKoSa4&s=Q7T-1zinvB8CyQKTLjBy8E5EFeBScmFX7T-5RJ8svHs&e=> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
