On Mon, Jul 9, 2018 at 4:57 PM Graeme Gillies <[email protected]> wrote:
> On 10/07/18 04:40, Gregory Farnum wrote: > > On Sun, Jul 8, 2018 at 6:06 PM Graeme Gillies <[email protected]> wrote: > >> Hi, >> >> I was wondering how (if?) people handle rotating cephx keys while >> keeping cluster up/available. >> >> Part of meeting compliance standards such as PCI DSS is making sure that >> data encryption keys and security credentials are rotated regularly and >> during other key points (such as notable staff turnover). >> >> We are currently looking at using Ceph as a storage solution and was >> wondering how people handle rotating cephx keys (at the very least, the >> admin and client.$user keys) while causing minimal/no downtime to ceph >> or the clients. >> >> My understanding is that if you change the keys stored in the ceph kv db >> then any existing sessions should still continue to work, but any new >> ones (say, a hypervisor establishing new connections to osds for a new >> vm volume) will fail until the key on the client side is also updated. >> >> I attempted to set two keys against the same client to see if I can have >> an "overlap" period of new and old keys before rotating out the old key, >> but it seems that ceph only has the concept of 1 key per user. >> >> Any hints, advice, or any information on how to achieve this would be >> much appreciated. >> >> > This isn't something I've seen come up much. Your understanding sounds > correct to me, so as a naive developer I'd assume you just change the key > in the monitors and distribute the new one to whoever should have it. > There's a small window in which the admin with the old key can't do > anything, but presumably you can coordinate around that? > > The big issue I'm aware of is that orchestration systems like OpenStack > don't always do a good job supporting those changes — eg, I think it embeds > some keys in its database descriptor for the rbd volume? :/ > -Greg > > > I think the biggest problem with simply changing keys is that lets say I > have a client connecting to ceph using a ceph.client.user account. If I > want to rotate the keys for that I can simply do that ceph cluster side, > but then I also need to do that on the client side (in my case virtual > machine hypervisors). DUring this window (which might be tiny with decent > tooling, but still non-zero) my clients can't do new connections to the > ceph cluster, which I assume will cause issues. > Well, it will depend on what they're doing, right? Most Ceph clients just set up a monitor connection and then maintain it until they shut down. Although I guess if they need to establish a session to a *new* monitor and you've changed the key, that might not go well — the client libraries really aren't set up for that. Hrm. > > I do wonder if an RFE to allow ceph auth to accept multiple keys for > client would be accepted? That way I would add my new key to the ceph auth > (so clients can authenticate with either key), then rotate it out on my > hypervisors, then remove the old key from ceph auth when done. > PRs are always welcome! This would probably take some work, though — the "AuthMonitor" storage would need a pretty serious change, and the client libraries extended to deal with changing them online as well. -Greg
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
