cyriltovena opened a new pull request, #66484:
URL: https://github.com/apache/doris/pull/66484
### What problem does this PR solve?
Issue Number: close #65432
Related PRs: #62788, #64766
Supersedes #65433 with a single clean commit on current `master`.
Problem:
GCP storage vaults currently require long-lived HMAC access keys. GKE
deployments using Workload Identity Federation have no supported way to
authenticate the vault data path with the pod's short-lived Google identity.
Solution:
Add `gcp_workload_identity`, a GCP-only storage-vault credentials provider.
BE and recycler obtain short-lived OAuth2 tokens from the GKE/GCE metadata
server and attach them to GCS XML API requests while leaving AWS SigV4 signing
disabled.
```sql
CREATE STORAGE VAULT gcs_vault PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_workload_identity"
);
```
The implementation:
- Adds `CredProviderTypePB::GCP_WORKLOAD_IDENTITY` for storage-vault
propagation. The legacy S3 resource/Thrift path remains unchanged.
- Uses only the metadata-server flow required by GKE Workload Identity; it
does not parse service-account key files or implement a partial ADC chain.
- Caches tokens process-wide, refreshes five minutes before expiry, retains
a still-valid token across transient refresh failures, and serializes
concurrent refreshes.
- Applies the bearer token to every BE and recycler GCS request, including
each listing page.
- Supports HMAC-to-Workload-Identity and Workload-Identity-to-HMAC ALTER
transitions while clearing obsolete credential fields.
- Returns an explicit `NotSupported` result for presigned URLs.
Upgrade compatibility:
The new persisted credential-provider enum is not understood by older BE or
recycler binaries. Do not create or alter a Workload Identity vault during a
mixed-version rollout; upgrade every BE and recycler consumer first. Explicit
HMAC or role credentials take precedence if a stale unknown enum value survives
a rollback transition.
Security Impact:
This removes the need to distribute long-lived HMAC keys on GKE. Workload
Identity is restricted to `provider = GCP` and the canonical
`https://storage.googleapis.com` endpoint at FE, Meta Service, BE, and recycler
boundaries. HTTP and arbitrary endpoints are rejected so OAuth bearer tokens
cannot be sent to an untrusted destination. Token acquisition failure fails
closed instead of falling back to anonymous access.
Testing:
- Added FE tests for provider/credential validation, HTTPS endpoint
canonicalization and rejection, storage-vault-only scope, and partial ALTER
mapping.
- Added Meta Service tests for creation, invalid provider/endpoint
combinations, credential transitions, and cleanup.
- Added BE tests for token caching, refresh, transient failure, concurrent
refresh, fail-closed behavior, authorization headers, client validation, and
typed presigned-URL errors.
- Added recycler authorization-header coverage.
- `git diff --check` passes.
- `protoc --proto_path=gensrc/proto --descriptor_set_out=/dev/null
gensrc/proto/cloud.proto` passes.
- Local C++ formatting/build and FE/BE unit execution are unavailable
because this checkout lacks LLVM 16, a Java runtime, and Doris's third-party
build bundle. PR CI must run the supported toolchain.
- End-to-end validation on GKE with Workload Identity remains required
before merge.
### Release note
Support keyless GCS storage vault authentication on GKE with Workload
Identity Federation using `"s3.credentials_provider_type" =
"gcp_workload_identity"`.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test added; execution pending PR CI
- [ ] Manual test
- [ ] No need to test or manual test
- Behavior changed:
- [x] Yes. Adds GKE Workload Identity authentication for GCP storage
vaults.
- Does this need documentation?
- [x] Yes. A doris-website follow-up will document the provider value
after review.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
