sollhui opened a new pull request, #70753:
URL: https://github.com/apache/doris/pull/70753

   ### Background
   
   Iceberg REST catalogs can return temporary storage credentials with table 
metadata. Doris must use those credentials for FE metadata access and BE data 
access, while the REST catalog owns the authentication needed to renew them.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   Catalog-issued `gcs.oauth2.token` credentials are filtered out or cannot be 
represented by the native GCS authentication path. Queries can consequently use 
the wrong storage identity or fail authentication. Support the supplied token, 
refresh it on FE using live catalog authentication, and send a storage-only 
snapshot to BE. The change spans the refactored filesystem, Iceberg and 
object-storage modules because credential selection, renewal, serialization and 
client lifetime must agree.
   
   ### Details
   
   - **Change map:** Production +923/-57 lines, tests +805/-1, build +47/-0; no 
generated files or documentation changes. About 435 production lines implement 
catalog-owned renewal and shared token state; the remaining changes carry 
tokens through FE/BE authentication, isolate BE snapshots, propagate 
normalization failures and release token-specific clients.
   - **Frequency:** Always for the reproduced token-only property binding; live 
Polaris/GCS queries were not exercised.
   - **Root cause:** Cloud-property filtering omits `gcs.*`, native GCS 
credentials lack an access-token mode, and storage normalization treats 
conversion errors as absent credentials.
   - **Why existing tests missed it:** Existing GCS authentication tests cover 
ADC, VM identity, impersonation and HMAC rather than catalog tokens. The 
closest external vended-credential regression targets S3/OSS/COS and is 
disabled for unavailable infrastructure; it does not exercise GCS renewal.
   - **Fix:** Recognize token aliases, include token/expiry in cache identity, 
retain per-table and delegated-user authentication during refresh, and close 
refresh state with the catalog. BE uses weak, bucket-isolated client caching 
for supplied tokens. Temporary tokens cannot become persisted resource/vault 
credentials.
   - **Other paths and branches:** Scan and write planning share credential 
extraction. JNI system-table serialization strips catalog/user secrets and FE 
context while preserving a current storage token. Other authentication modes 
keep their existing cache behavior. Release branches and the introducing commit 
were not established; this targets master only.
   - **Behavior change:** Invalid vended credentials now fail instead of 
falling back to static catalog credentials, including normalization failures 
for other providers. No new configuration is required. BE does not renew a 
running query's token; expiry can still fail that query.
   
   ### Testing
   
   - Locally passed 69 tests in the GCS and shared Iceberg modules: 13 
resolver, 8 authentication, 26 filesystem-property and 22 refresh/lifecycle 
cases, using Iceberg 1.12.0 and a local HTTP service.
   - An isolated SDK harness passed the 22 refresh cases plus the actual 
system-table FileIO serialization test. The latter checks that only the storage 
token survives transport.
   - The same minimal resolver assertion fails on unmodified master and passes 
on this change. Production C++ credential parsing compiled with C++20 and 
passed 26 boundary cases.
   - Added BE weak-cache/parser tests and FE bridge tests; these have not run. 
Full FE/BE builds, full connector tests, mixed-version execution and live GCS 
integration remain unverified because generated sources and third-party build 
dependencies are unavailable locally.
   
   ### Notes for Reviewers
   
   Renewal performs outbound FE HTTP requests to catalog-configured or 
server-vended credential endpoints. Setting a catalog endpoint requires 
CREATE/ALTER CATALOG privileges; the user-configured refresh endpoint is 
registered with the existing conditional SSRF gate. Review the REST session 
adapter and credential snapshot boundary first. FE filesystem code remains 
independent of Iceberg.
   
   ### Release note
   
   Support Iceberg REST catalog-issued GCS OAuth2 tokens and FE renewal for 
subsequent queries. Invalid vended credentials now report errors instead of 
silently falling back to another storage identity.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; document GCS vended-credential support, 
renewal and running-query expiry limitations in the Iceberg REST catalog guide.
   - Pre-submission status: Partially verified: full FE/BE and connector 
integration tests, mixed-version operation and live GCS integration remain 
unverified.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code adaptation, tests and this description.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to