MitchDrage opened a new pull request, #14170:
URL: https://github.com/apache/cloudstack/pull/14170
## Description
Object storage credentials are provisioned per CloudStack account today:
every bucket an account owns shares one access/secret key pair, and there is no
way to rotate it.
This PR introduces per-bucket keys, a rotation mechanism, and the migration
from per-account to per-bucket keys.
Fixes #14167
Discussed on dev@: "[DISCUSS] Per-bucket credentials and key rotation for
object storage".
This PR's changes:
- Each bucket gets its own credential on the backend, so a leaked key
exposes one bucket rather than every bucket the account owns.
- Each credential holds two independent key slots, so a key can be rotated
while consumers still use the other one, and revoked when they have moved.
- Existing buckets keep working exactly as they do now. Nothing is migrated
automatically and the schema change is additive.
- An administrator moves an account onto per-bucket credentials explicitly,
per object store, then moves each bucket, then rotates the account key to
finish. Existing keys keep working throughout.
- On Ceph this is built on RGW accounts, so it needs Ceph Squid or later.
Older gateways, and providers that do not offer this, keep today's behaviour
and say so rather than half-working.
- New APIs: `rotateBucketKey`, `revokeBucketKey`, `migrateBucketCredential`,
`migrateObjectStoreAccount` and `rotateObjectStoreAccountKey`. `listBuckets`
and `listObjectStoragePools` gain fields and filters for the new state.
- New UI: a Keys tab per bucket, a Credential Scope column and filter, a
migrate action on buckets, an Object Storage tab on accounts, and a readiness
line on the object store itself.
- Details of storage backend's are only shared with the root
administrator, e.g. backend type, versions, overall-readiness for migration.
General users are only told that the backend doesn't support per-bucket
credentials.
- MinIO, Cloudian and the Simulator are untouched and report that they do
not offer per-bucket credentials.
### How it works
- Two new tables, `bucket_credential` (one dedicated backend identity per
bucket) and `bucket_credential_key` (one row per key slot, secret encrypted
with `@Encrypt`). A bucket without a `bucket_credential` row behaves exactly as
today (account-scoped key) - the schema change is additive and no data is
migrated.
- The existing `bucket.access_key`/`bucket.secret_key` columns become a
mirror of the bucket's newest active key, so every existing consumer of them
(`BucketTO`, the policy/versioning driver calls, the UI object browser) keeps
working unchanged.
- New provider-agnostic driver operations on `ObjectStoreDriver`
(`createBucketCredential`, `createBucketCredentialKey`,
`removeBucketCredentialKey`, `deleteBucketCredential`, plus the account-level
`supportsBucketCredentials` / `accountSupportsBucketCredentials` /
`migrateAccountForBucketCredentials`). `BaseObjectStoreDriverImpl` provides
"unsupported" defaults, so MinIO and Cloudian are untouched; the Simulator gets
a stateless fake so the service layer is exercisable in CI.
- New APIs: `rotateBucketKey`, `revokeBucketKey`, `migrateBucketCredential`
(user-level, bucket-scoped) and, for root/domain admins,
`migrateObjectStoreAccount` and `rotateObjectStoreAccountKey`. The latter
completes a migration: once no bucket of the account on a store still uses the
account key, it issues CloudStack a fresh account key and revokes the old one,
so a shared key copied before the migration no longer opens every bucket. It is
refused while legacy buckets remain, and is never run automatically.
`listObjectStoragePools` accepts `accountid` and then reports, per store,
whether the store supports per-bucket credentials at all, whether the account
is migrated and how many buckets still use the account key. `listBuckets`
responses gain `credentialscope` (`bucket`/`account`) and a `keys` list, and
accept `credentialscope` as a filter so the buckets still using the account
credential can be listed directly; secret keys are marked sensitive for API-log
redaction.
- Global setting `object.storage.per.bucket.credentials` (default `true`)
applies **only** when an account is first given an identity on an object store.
It never changes an account that already has one:
| The account on that store | `true` (default) | `false` |
|---|---|---|
| has no identity yet | is created so that each of its buckets gets its own
credential | is created sharing one credential across its buckets, as in
earlier releases |
| shares one credential | unchanged - moving to per-bucket credentials is
always an explicit admin action | unchanged |
| has been migrated | each new bucket gets its own credential | the same:
buckets keep getting their own credentials. Sharing one here would mean writing
the account's root key onto a bucket, where every user of the account can read
it, which undoes the migration |
- To drive an account's object storage, CloudStack has to keep a credential
of its own for that account on each store: the RGW account id and the account's
root key. Account details are the natural place to keep it, but they are handed
back to API callers as `accountdetails` in `listAccounts` responses, and this
credential is CloudStack's rather than the account's. Storing it there as-is
would therefore publish it, which would defeat the migration: the root key
opens every bucket the account owns, and the point of rotating it at the end of
the migration is that nobody else holds it.
So these rows are named with a reserved prefix, `objectstore-`, held as a
constant (`ObjectStore.ACCOUNT_DETAIL_PREFIX`) so that the driver writing them
and the API layer agree on it, and `ApiDBUtils.getAccountDetails` drops/hides
rows carrying that prefix before the response is built. The secret is
additionally encrypted with `DBEncryptionUtil`. The namespace is the only thing
removed; every other account detail, including those existing providers already
store, is returned exactly as before.
- UI: a Keys tab per bucket, a Credential Scope column and filter, a migrate
action on buckets, an Object Storage tab on accounts, and a readiness line on
the object store itself. Each screen is listed under "What changes in the UI"
below.
### Points a reviewer may want to ponder on
These are deliberate choices I've made along the way:
- I've made all references to versions (API's, etc) to the 24.0.0 version
based on the vote to drop the '4'. Please also let me know if this need to go
in a version other than 24.
- **Ceph Squid or later only.** Per-bucket identities on plain RGW users
fragment object ownership, which breaks public bucket policies and leaves the
old key with access to old objects. RGW accounts fix the cause but arrived in
Squid. Older gateways keep today's behaviour.
- **Account migration cannot be undone.** Adopting a user into an RGW
account is permanent at the backend, so the action is explicit, admin-only and
warned about, and CloudStack never does it implicitly.
- **The global setting changed meaning.** It now decides how an account is
set up the first time it uses a store, and no longer forces a migrated account
back onto a shared credential. Without that, turning the setting off would
quietly undo a completed migration.
- **Shared UI code is touched.** List sections gain an `optionalColumns`
option, and a details list can now vary per record. Both are additive and inert
elsewhere, but they are shared files.
- **Account details are filtered in a shared response path.**
`ApiDBUtils.getAccountDetails` now strips the reserved namespace this feature
writes. Nothing else is removed, but every account response goes through that
method.
- **Secrets are still returned by the API.** A bucket's keys appear in
`listBuckets` for callers who pass its access check, as the account-scoped key
does today. Marking them sensitive keeps them out of logs; hiding them from the
response would break the object browser and existing automation.
- **A bucket's own identity policy grants `s3:*` on that bucket**, which
includes changing the bucket's policy. It is confined to the bucket the key
belongs to, but a narrower action list is arguable.
- **An interrupted rotation can leave an untracked key**, and rotation
reports it rather than deleting it, because CloudStack did not create it and it
may be an operator's own.
- **`listBuckets` gained a per-bucket gateway call** for buckets still on
the account credential of a migrated account, to decide whether the migrate
action applies. Accounts with no migration record answer from the database
alone.
- **An existing parameter starts working.** `listBuckets`'s
`objectstorageid` has been accepted and ignored since 4.19.0; anything relying
on it returning every bucket will see a filtered list.
- **Why a store cannot support the feature is shown only to root admins**,
on the object store itself, because the causes name the gateway's own admin
credential and its capabilities. A domain admin refused a migration is told to
contact their platform administrator instead.
- **The driver contract gained an optional explanation**
(`bucketCredentialsUnsupportedReason`). It is a default method, so no provider
has to implement it, but adding a method to `ObjectStoreEntity` means an
incremental build has to rebuild the implementing module.
<details>
<summary><b>UI changes people will see</b></summary>
**Buckets list**
- A **Credential Scope** column, reading "Per-Bucket" or "Account". It stays
hidden until the viewer can see at least one bucket with its own credential,
then appears on its own; after that the column picker decides.
- A **Credential Scope** filter in the search panel, with the same two
values.
**A bucket's Details tab**
- **Credential Scope** is shown.
- Once a bucket has its own credential, the access key and secret are no
longer listed here. They move to the Keys tab, so there is one place to find
them. A bucket still on the account credential is unchanged.
**A bucket's Keys tab (new, only on buckets with their own credential;
viewable/usable by the bucket owner, admins and domain admins)**
- Two rows, one per key slot, each showing its state, its access key with a
copy button, its secret masked behind a reveal, and when it was created.
- Per row: rotate an active slot, create a key in an empty slot, or revoke.
Revoke is disabled on the last active key, with a tooltip saying why.
- Each action confirms first, in the standard dialog.
**A bucket's actions**
- **Migrate to Per-Bucket Credential (new, viewable/usable by the bucket
owner, admins and domain admins)**, on buckets still using the account
credential. It only appears once the owning account has been set up on that
store, so it cannot be started in a state that would be refused.
**An account's Object Storage tab (new, viewable/usable by admins and domain
admins)**
- One card per object store, supported stores first.
- While a store has work outstanding: a four-stage progress display (account
key, migrate buckets, rotate account key, complete), the count of buckets still
on the account credential, a **Migrate Account to Per-Bucket Credentials**
button, a **Show Buckets** link filtered to that store's remaining buckets,
and, once every bucket has been moved, a warning that the account key still
opens everything with a **Rotate Account Key** button.
- When a store is finished: a single line saying per-bucket credentials are
in use, with no stages and no buttons.
- When a store cannot support the feature: a plain notice saying so and to
contact the platform administrator. No backend or version or other information
is named in order to hide platform internals from users.
- The explanation of the migration only appears while some store still has a
stage left. After is has been migrated, the steps to migrate are replaced by a
note saying that it's been migrated.
**Infrastructure, Object Storage, a store's Details tab (root admins only)**
- **Per-Bucket Credentials: Ready** or **Not ready**.
- When not ready, **To Resolve Before Migration** naming the cause: a
missing `info` or `accounts` capability on the store's admin credential, a
gateway that predates RGW accounts, a provider that does not offer the feature,
or an unreachable admin API.
**Events**
- Key rotation, key revocation, bucket credential migration, account
migration and account key rotation each raise their own event, visible on the
bucket's and account's Events tabs.
</details>
<details>
<summary><b>Changes to shared UI code, and why they are needed</b></summary>
Four files outside this feature are touched. Each is additive and gated so
they don't apply to any other section, but they are shared so they are listed
here for completeness.
| File | Change | Effect elsewhere |
|---|---|---|
| `views/AutogenView.vue` | a section may declare `optionalColumns`: columns
that stay unticked until a condition first holds, then are ticked once and left
to the viewer thereafter | none unless a section declares it; the method
returns on its first line when the route has no such declaration, and only the
buckets section has one |
| `config/router.js` | carries that declaration into the route metadata,
beside the existing `columns` | none; the property is only copied when present |
| `components/view/DetailsTab.vue` | a `details()` function is now called
with the record: `details(this.resource)` | none; twelve of the thirteen
existing definitions take no arguments and ignore it |
| `components/view/ListView.vue` | renders a `credentialscope` cell through
the translation table, so it reads "Per-Bucket" rather than `bucket` | none;
the branch is keyed on a column name no other section uses |
| `components/view/SearchView.vue` | `credentialscope` added to the list of
filters drawn as a dropdown, with its two options | none; both additions are
keyed on that filter name |
Why they are needed rather than done in the section config:
- **The new column in the bucket list is hidden until it's needed** - This
cannot be expressed in a section config today: the column list is fixed, and
every column in it is selected by default. A column stating that every bucket
has "Account" credentials on every row until the first migration is noisy. I
decided that showing it when one or more of the buckets has per-bucket
credentials is cleaner and more useful. It is not specific to this feature, so
it was added as a general option rather than a special case for buckets.
- **A bucket's Details tab drops the key fields once that bucket has its own
credential** - This is likewise not expressible today: `details` may be a
function, but it was called with no arguments, so it could vary per route and
not per record, and this needs to vary for one bucket and not another. Passing
the record is backwards compatible, since an existing zero-argument function
ignores it.
- **The Credential Scope search filter is a dropdown of the two values
rather than a free-text box** - This cannot be expressed either: a section
names its filters, but whether one draws as a text box or a dropdown is decided
by a list of names inside `SearchView.vue`.
- **The column reads "Per-Bucket" and "Account" rather than the raw values**
- This one could have been done in the section config: a column may be given as
an object carrying a render function, as `plugin/quota.js` does. I put it in
`ListView.vue` because that mechanism replaces the row's value with the
rendered text, so the record would then carry "Per-Bucket" where other code
reads `bucket`. It is a trade-off, and I will move it if reviewers prefer the
config route.
</details>
<details>
<summary><b>Ceph RGW implementation - requires Ceph Squid (v19) or
later</b></summary>
On plain RGW users (the existing CloudStack approach to RGW credentals),
giving an existing bucket its own identity cannot move the ownership of the
objects already in it. The admin REST API can relink a bucket to another owner,
but only the `radosgw-admin bucket chown` command rewrites the objects inside.
CloudStack accesses the gateway over REST and S3 with no shell access to it, so
it has no way to migrate ownership. That leaves migrated buckets with split
ownership: public bucket policies would 403 on the old objects and the old key
would retain access to them. RGW **accounts** remove the cause: the account
owns every bucket and object regardless of which identity wrote it. So the Ceph
driver maps:
- one RGW account per CloudStack account **per object store** (an account
can be migrated on one store and still be legacy on another; the state is kept
per store in `account_details`), whose root user is the existing account-UUID
RGW user;
- one IAM user per bucket, with an inline policy scoped to that bucket and
its access keys mapping onto the two key slots;
- a bucket-policy statement naming the IAM user, so it can also reach
objects the account root wrote before the account migration (identity policies
alone cannot; verified on 20.2.4). CloudStack's public/private bucket policy is
regenerated together with that statement so the two features compose.
CloudStack's records can disagree with the gateway: a database restored from
another environment, or a store re-pointed at a different cluster, leaves
CloudStack believing an account was migrated when that cluster has never heard
of it. The driver therefore asks the gateway whether the account's user really
is an account root there before any account-level operation.
An account counts as unmigrated only when the gateway answers and says so.
An unreachable admin API, a 403 while an admin credential's capabilities are
being changed, or a probe that flaps during a rolling gateway upgrade all leave
the migration record standing, since migration is permanent at the backend.
Treating those as unmigrated would drop the account back to its shared
credential and write the account root key onto its next bucket. Where a
per-bucket credential genuinely cannot be issued, bucket creation fails with
the gateway's error.
Existing accounts are **never migrated implicitly**: adopting a user into an
RGW account is permanent, so it takes an explicit `migrateObjectStoreAccount`
by a root or domain admin. Until then the account keeps today's behaviour, new
buckets included. A brand-new account on a store that supports the feature is
set up for it directly.
The Ceph plugin gains a dependency on `aws-java-sdk-iam` (already used by
the Cloudian driver) and a small SigV2 client for the `/admin/account` and
`/admin/info` endpoints that radosgw-admin4j does not cover.
Verified against three RGW releases (single-container clusters, same script):
| | Reef 18.2.7 | Squid 19.2.3 | Tentacle 20.2.4 |
|---|---|---|---|
| `/admin/info` | 200 | 200 | 200 |
| `accounts` capability | cannot be granted | granted | granted |
| `POST /admin/account` | 405 | works; duplicate → 409
`AccountAlreadyExists` | same |
| `GET /admin/account` | 405 | 403 whatever capabilities are granted, as
does DELETE | 404/200 as expected |
| adoption, IAM users/keys/policies, bucket-policy grant | n/a | identical
to Tentacle | verified |
Reef cannot support this at all: its account endpoint answers 405, and
`radosgw-admin caps add` refuses the `accounts` capability, so no credential on
it could hold what the endpoint requires. Squid is the floor.
The driver never looks an account up or deletes one, since Squid answers 403
to `GET` and `DELETE /admin/account` whatever capabilities the credential
holds. The account id is derived from the CloudStack account UUID (`RGW`
followed by 17 digits), so the same account always produces the same id, and
`createAccount` treats a 409 as success.
An RGW account root is not a listed IAM user: it never appears in
`ListUsers`, and IAM calls that name it are rejected. `rotateAccountKey`
manages its keys with the no-`UserName` form of `CreateAccessKey` and
`DeleteAccessKey`, authenticated as the root itself, which is how an AWS
account's own root credentials work.
The plugin registers its own signer (`RgwIamSigner`) for IAM calls. The AWS
Java SDK signs the request before the HTTP layer attaches the form
`Content-Type`, so that header never reaches the signature, and Tentacle
answers 403 to a request whose content type was unsigned.
CloudStack works out whether a store supports this by asking the gateway:
`/admin/info` must answer, the account endpoint must not return 405, and the
store's admin credential must hold **both** `info` and `accounts` (with
`accounts` alone the gateway answers 403 on `/admin/info`). The three causes
are indistinguishable from outside, so a root admin refused a migration is told
which one applies, for example "the object store's admin credential is missing
the 'info' capability". The probe re-checks automatically, so upgrading a
gateway is picked up with no CloudStack configuration change.
</details>
### When things are interrupted or the backend changes
Every step that touches the gateway is repeatable. CloudStack records a
migration once the gateway work is done, so a management server that dies
mid-way leaves the account working. Running the same action again finishes the
job.
- **Interrupted account migration.** The RGW account is created (idempotent
on 409), the user is adopted, and only then is the migration recorded. Fail
before that record and CloudStack still sees a legacy account: its buckets keep
working on the key they already had, which adoption preserves, and the
bucket-level migrate action stays hidden. Running `migrateObjectStoreAccount`
again re-adopts harmlessly and records the result. Verified by deleting those
records after a real migration and retrying.
- **Interrupted bucket credential.** The IAM user, its policy and its key
are created before anything is written down. A retry re-uses the existing user
and clears any access key CloudStack is not tracking on it, so a half-finished
attempt cannot leave a key nobody knows about. That user belongs to the bucket
alone, which is what makes clearing it safe.
- **Interrupted account key rotation.** The new key is created, recorded,
and only then is the old one removed, so CloudStack always holds a key that
works. If recording fails, the new key is deleted again and the old one stays
in force.
- **What a bucket's users can see while it is being created.** Creating a
bucket is an asynchronous job, and the bucket row is listable from the moment
it is allocated. The driver therefore never writes the account key onto the row
of a bucket whose account is set up for per-bucket credentials, so the
account's root key does not pass through a place every user of the account can
read, and responses withhold a bucket's keys until it reaches the `Created`
state.
- **Logs.** The admin API client never logs a user record, since those carry
the user's secret keys; only the method, path and status are traced.
- **Two operations at once.** Account migration and account key rotation
take a lock for that account on that store, so two administrators, or two
management servers, cannot both issue a key and then revoke each other's. The
gateway calls made while that lock is held are bounded — a 20 second socket
timeout, one retry, and a 30 second ceiling on each call — so an unresponsive
gateway releases the lock in about a minute. The AWS SDK's defaults would have
allowed ten.
- **The gateway changes underneath.** A support probe that starts failing,
an admin credential that loses the `accounts` capability, or an unreachable
admin API cannot un-migrate an account: its record stands and its buckets keep
getting their own credentials. Operations that genuinely need the gateway fail
with the gateway's error rather than falling back to the shared credential.
### Incidental fix
`listBuckets` has accepted an `objectstorageid` parameter since 4.19.0 that
seems to have never worked. It declared `entityType =
StoragePoolResponse.class` (primary storage), so the UUID could never resolve
and the call failed with "entity does not exist"; and
`searchForBucketsInternal` never read the parameter at all, so once the UUID
did resolve every bucket was still returned. Both are fixed here: the entity
type is corrected to `ObjectStoreResponse.class`, and the search now applies
the filter. The account's Object Storage tab relies on it for its "Show
Buckets" link (root admin only, matching the parameter's existing
authorization, which is unchanged).
I haven't found an issue raised against this, but I needed this fixed for my
tests to work so I have fixed it here.
### Documentation
Operator documentation is being raised separately against the documentation
repository, and will be linked here once it is up. It covers the capabilities
the store's admin credential needs (`info` and `accounts`), what account
migration does and that it cannot be undone, and the stages an administrator
works through.
### Open question: how long should the account's Object Storage tab existin
the Account page?
The tab, and the migration it drives, exist only because deployments have
accounts that predate per-bucket credentials, and because some object storage
cannot support them. Both shrink over time. Once the providers CloudStack
supports offer per-bucket credentials, and versions that cannot are out of
support, there is a decision to make: deprecate support for the backends and
versions that cannot do this (Ceph before Squid is already end of life), at
which point new accounts are always per-bucket, the migration becomes a one-off
upgrade concern, and this tab can go. One suggestion I could make is to add the
Object Storage provider and version into the anonomised usage telemetry service
that has been discussed recently. That way, when we have a view on how much
Ceph Reef is out there, we can make an informed decision on when to stop
supporting it.
### Known limitations / follow-ups
- `bucket.secret_key` still holds the mirrored active secret in clear, as it
does today; only the new `bucket_credential_key` table is encrypted. Encrypting
the existing column would mean migrating the rows already in it, and a
rolling-upgrade window in which management servers disagree about the format,
so changing it is left out of this PR. It would also not change what the API
hands out, since that secret is already returned to callers who can see the
bucket and changing that may be a breaking change.
- Existing provider account details, such as the account-scoped keys the
Ceph and MinIO drivers have always written, are left exactly as they are; only
the details this feature adds use the reserved namespace.
- A rotation interrupted between issuing a key and recording it can leave a
key on the account's root user that CloudStack does not track, and which still
opens every bucket of the account. Rotation removes only the key it issued, and
logs a warning naming any others rather than deleting credentials it did not
create, since one of them may be an operator's own. Reviewing them is a manual
step on the gateway.
- Bulk migration of buckets and MinIO/Cloudian implementations are
follow-ups. I don't have familiarity with those software packages nor an
environment to test them on.
- Objects written before an account is migrated stay owned by its
pre-migration user, and the admin API cannot change that. CloudStack covers
them by granting the bucket's credential in the bucket policy as well as its
identity policy. A bucket policy written straight to the gateway drops that
grant, leaving the credential blind to everything from before the migration;
changing the bucket's public/private setting in CloudStack restores it.
## Types of changes
- [ ] Breaking change
- [x] New feature
- [ ] Bug fix
- [ ] Enhancement
- [ ] Cleanup
## Feature/Enhancement Scale or Bug Severity
- [x] Major
## Trying it out
Documentation is being raised separately, so here is enough to exercise the
feature.
**What you need.** A Ceph cluster running Squid (v19) or later with RGW. A
single container is enough. The RGW admin user whose keys you register the
object store with needs two capabilities, and it will look unsupported without
both:
```
radosgw-admin caps add --uid=<admin uid> --caps="info=*;accounts=*"
```
Register it in CloudStack as a Ceph object store in the normal way.
Infrastructure -> Object Storage -> the store's Details tab reports whether it
can provide per-bucket credentials, and what to resolve if it cannot. That
readiness information is root admin only.
**A new account (the short path).** An account that has never used this
store is set up for per-bucket credentials on its first bucket, with no
migration step.
1. Create a bucket in that account. Storage -> Buckets shows Credential
Scope "Per-Bucket".
2. Open the bucket -> Keys tab. Slot 1 holds a key, slot 2 is empty.
3. Use the slot 1 key with any S3 client. It reaches this bucket and is
denied on another bucket in the same account.
4. Rotate the key. Slot 2 fills with a different key. Both keys now work.
5. Revoke slot 1. That key is rejected by the gateway; slot 2 keeps working.
6. Try to revoke slot 2. It is refused, because a credential keeps at least
one active key.
7. Delete the bucket. `radosgw-admin user list` shows the bucket's IAM user
is gone.
**An existing account (the migration path).** This is the path that matters
for upgrades, and it is what the account's Object Storage tab walks an
administrator through. Use an account that already has buckets on the store.
1. Account -> Object Storage tab. It shows the store, the account's state on
it, and the next step.
2. **Migrate the account.** This adopts the account's existing RGW user as
the root of a new RGW account at the gateway. It cannot be undone (the UI warns
about this). The existing keys keep working afterwards: check with an S3 client
before and after. Existing buckets still read "Account" for Credential Scope.
3. **Give each bucket its own credential.** "Show Buckets" from the tab
lists the account's buckets on this store that still share the account key.
Each has a Migrate to Per-Bucket Credential action. A migrated bucket's own key
can read objects written before the migration.
4. **Rotate the account key.** Offered once no bucket is left on the account
key, and refused before that with the list of buckets still using it.
Afterwards the original account key is rejected by the gateway, and the
per-bucket keys are unaffected.
**An unsupported store.** Register a Reef (18.x) cluster, or drop the
`accounts` capability from the admin user of a Squid one. The account's Object
Storage tab says the store cannot be migrated. As root admin, the store's
Details tab names the cause; a domain admin is told to contact their platform
administrator.
**Checking at the gateway.**
```
radosgw-admin account list # one account per
migrated CS account
radosgw-admin user list # an IAM user per
migrated bucket
radosgw-admin bucket stats --bucket=<name> # owner is the account id
```
**Turning it off.** `object.storage.per.bucket.credentials=false` makes new
accounts use the shared credential as before. Accounts already migrated stay
migrated.
## How Has This Been Tested?
- Unit tests: `CephObjectStoreDriverImplTest` (37) and
`BucketApiServiceImplTest` (26), plus a new `ObjectStoreAccountDetailTest` (3)
and four cases added to `QueryManagerImplTest` for the bucket filters.
- RGW behaviour verified with scripted checks against single-container Ceph
18.2.7 (Reef), 19.2.3 (Squid) and 20.2.4 (Tentacle) clusters: account probe,
adoption keeps legacy keys working, IAM user/policy/key lifecycle, policy
scoping, public-policy coverage of IAM-written objects, bucket-policy grant for
pre-adoption objects, revocation, no two-key limit.
- End-to-end through a management server (simulator profile) with the Squid
and Tentacle containers registered as Ceph object stores:
- Tentacle, new account: create bucket -> dedicated credential,
account-owned bucket, IAM user with one key, key isolated to its bucket; rotate
into slot 2; revoke slot 1 (RGW rejects the old key); last-key revoke refused;
rotate back into slot 1; delete bucket removes the IAM user. 19/19.
- Squid, legacy account (seeded with
`object.storage.per.bucket.credentials=false`): `migrateObjectStoreAccount`
adopts the RGW user as account root with its keys still valid;
`migrateBucketCredential` gives the existing bucket a dedicated key that reads
the pre-migration object; then the same new-bucket sequence as above. 25/25.
- Ceph 20.2.2, legacy account with a pre-migration object: account
migration, bucket migration (bucket key reads the old object),
`rotateObjectStoreAccountKey` refused while a legacy bucket remains, then
succeeds once none remain - old account key rejected by RGW, root user holds
one new key, per-bucket keys unaffected, new buckets created with the rotated
key are per-bucket, a second rotation succeeds. Per-store scoping verified: the
same account reports legacy on the other two stores.
- Marvin: a `test_02_bucket_key_rotation` case added to `test_bucket.py`,
with the helpers it needs added to `marvin/lib/base.py`. It creates a Simulator
object store, then checks that a new bucket has one active key slot mirrored
onto the bucket, that rotation fills the free slot with a different key and
moves the mirror, that revoking leaves the other slot active, and that revoking
the last active key is refused. Run against a management server on the
simulator profile: both cases in the file pass.
- That file's existing `test_01_create_bucket` was failing before this PR,
since `createBucket` has a required `quota` parameter (from #10017 ?) that
wasn't filled. The one-line fix is included here, as our own case was written
from the same pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
