Thanks for the thoughts on the topic. I'm happy to start drafting a design so we can document decisions and have something visual to work on. Should I start something in confluence?
-----Original Message----- From: Wido den Hollander <[email protected]> Sent: Thursday, 27 August 2026 5:56 PM To: [email protected]; Mitchell Drage <[email protected]> Subject: Re: [DISCUSS] Per-bucket credentials and key rotation for object storage This email comes from outside the organization. Do not click links or open attachments unless it is an email you expected to receive. Op 25-08-2026 om 02:27 schreef Mitchell Drage: > Thanks for the feedback Wido. > >> It depends on how you look at this. Inside Ceph's RGW it's normal >> that there is a user who has multiple buckets [...] The same applies >> to a CloudStack account [...] If you want to separate, you can create >> multiple accounts in CloudStack, correct? > > I think there's a distinction to make between accounts owning resources that > are within Cloudstack, vs Access/Secret keys for S3 buckets where the > intention of those keys is to use them externally to the platform - i.e. in a > backup tool, a CI/CD tool, a third-party or SaaS app. These things are > typically outside of the span of control of Cloudstack or the administrator > that's interacting with Cloudstack itself, so to share these credentials > across buckets is the concern for me. Each bucket is typically for a > different tool with different supplier/supply chain risks, therefore its own > security boundary. I can't disagree with you there. I would think that it does make sense, but then you also want to be able to rotate the keys of buckets, that is a very valid use-case as well. > Yes, using separate accounts in Cloudstack to get different secrets is an > option, but I feel it's quite a heavy and inflexible requirement and > conflates tenancy with storage-specific requirements. > Ideally, I would see this being some form of IAM that could handle this, but > I don't see that being a nearby feature. Agree with you there. > >> Wouldn't this overload the backend? You keep creating users and I >> don't know how that scales and if both Minio and Ceph were designed for this. > > Fair question, and I think you'd know the RGW internals far better than I > would. My understanding is that RGW users are stored in metadata, and > production deployments run into the thousands of users without issue. Buckets > and users are stored in the metadata pool and just consume space rather than > memory, and don't materially impact performance. True, I don't think Ceph will have a real problem in handling this. > > If proliferation is a concern, perhaps one option would be to make per-bucket > identity opt-in per bucket rather than the default, so an identity is only > created when an administrator actually wants to isolate or delegate a bucket. > The default stays account-level and nothing changes for the common case. > > The last part to cover - we don't have any ability to rotate keys right now, > so at a minimum I feel there's a need for a second key and the ability to > revoke/rotate regardless of where we land on identity. > I know for Ceph at least, it's just a matter of adding a second key to the > existing RGW user and having Cloudstack handle these changes. True. The longer I think about it, the more it makes sense to have unique keys per bucket. How we achieve this technically and per backend, that's the next step, but for now this seems reasonable and logic to do. And key rotation should be there as well :-) Wido > > > Mitch > > > > > -----Original Message----- > From: Wido den Hollander <[email protected]> > Sent: Monday, 24 August 2026 10:33 PM > To: [email protected]; Mitchell Drage > <[email protected]> > Subject: Re: [DISCUSS] Per-bucket credentials and key rotation for > object storage > > This email comes from outside the organization. > Do not click links or open attachments unless it is an email you expected to > receive. > > > Op 24-08-2026 om 06:17 schreef Mitchell Drage: >> Hi all, >> >> I'd like to propose an improvement to the object storage feature and get >> feedback before I start to think about developing something. >> > > Thanks for this! Discussions and feedback a great. > >> >> Current behaviour >> >> Object storage credentials are provisioned per CloudStack account, not per >> bucket. The first bucket creates a backing user (e.g. a radosgw-admin user >> on the RGW provider) and its access/secret key is stored against the >> account. Every subsequent bucket in that account reuses the same key pair. >> Deleting and recreating a bucket returns the same keys, and there's no API >> to rotate or regenerate them. >> >> >> Problem >> >> - A leaked key exposes every bucket the account owns, not just one. >> - You can't delegate access to a single bucket without handing over access >> to all of them. >> - There's no safe way (or any way at all, from what I can see) to rotate a >> credential, since there's only one and revoking it breaks every client at >> once. >> > > It depends on how you look at this. Inside Ceph's RGW it's normal that there > is a user who has multiple buckets, with the Access and Secret key of that > user you can access the buckets. > > This is very similar to other S3 services. The same applies to a CloudStack > account, if you gain access to that account, you gain access to all the > Instances of that user. > > If you want to separate, you can create multiple accounts in CloudStack, > correct? > >> >> Proposal >> >> 1. Per-bucket credentials: each bucket gets its own backing identity rather >> than sharing the account's. This contains the impact of a leaked key to a >> single bucket. > > Wouldn't this overload the backend? You keep creating users and I don't know > how that scales and if both Minio and Ceph were designed for this. > > I'm not convinced yet, but I do appreciate feedback on features! > > Wido > >> 2. Two key pairs per bucket, with rotate and revoke: two independent slots >> so an operator can rotate one while clients continue on the other, then cut >> over and revoke the old one. This mirrors Azure's two-key model and AWS's >> two-access-keys-per-user. >> >> I'd keep the backing primitive as plain users (RGW/MinIO/Cloudian all >> support multiple keys per user), rather than the newer IAM-account >> constructs, to keep the change contained and provider-agnostic. >> >> >> Compatibility / migration >> >> - Existing buckets are untouched on upgrade. The bucket gains a nullable >> credential association; when null, it falls back to the account key exactly >> as today. >> - New buckets get their own credentials by default. >> - Operators opt in to migrating existing buckets via an explicit action >> (single and bulk), so they choose when any disruption happens rather than >> the upgrade forcing it. >> - Every supported S3 backend would need code changes which would be larger >> than I can take on practically, but also because I only have access to Ceph. >> I could develop for Ceph, and the others would need to be a roadmap item. >> Alternatively, other people familiar with those systems could develop code >> and test. Ideally everything would move together, but I'm not confident that >> we could have someone contribute for the MinIO and Cloudian backends, and I >> also see that there's an open PR for Dell ECS support >> (https://github.com/apache/cloudstack/pull/12124), so that would need >> updating as well. >> >> >> Future direction (I'm proposing it be out of scope here, for now) >> >> The longer-term destination might be an IAM / service-account style model >> where identities are decoupled from buckets and bound via policy, which is >> where AWS, Azure, GCP and now Ceph (accounts in Squid) all sit. >> That requires a significant amount of change in CloudStack and the >> introduction of IAM and granular policy, so I'm deliberately not tackling it >> now. This proposal is designed as a step towards it rather than against it: >> once credentials are their own entity, a many-to-many with permissions is an >> extension rather than a rewrite. >> >> Rough scope is a moderate amount of change touching the object store driver >> interface, the ceph provider plugin (ideally all the three provider >> plugins), the bucket schema, a new API command + UI tab, and an upgrade path. >> >> Does this seem like a reasonable direction? >> >> >> Thanks, >> Mitch >
