sollhui opened a new pull request, #70657:
URL: https://github.com/apache/doris/pull/70657

   ### Background
   
   BE and Recycler share provider-specific object-storage configuration and SDK 
clients after #70443. Azure currently accepts SharedKey through legacy S3-style 
properties; those properties cannot represent vended SAS or Entra ID client 
credentials.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70443 (prerequisite, cherry-picked as the first commit), #70386 
(typed Azure design reference)
   PR Type: Enhancement
   
   Problem Summary: Give data-lake connectors one typed FE-to-BE Azure access 
contract supporting SharedKey, SAS and OAuth2 client credentials. Apply it 
throughout reads and writes, including Iceberg delete files and deletion 
vectors. FE catalog properties and credential vending will use this interface 
in a follow-up; this PR implements the protocol and BE/shared storage side.
   
   ### Details
   
   - **Change map:** The implementation commit adds/removes 1,238/132 
production lines across 41 files, 817/9 test lines across 8 files, and 69 
documentation lines. Most caller changes only forward the optional 
configuration. No generated files are committed. The first commit is the 
standalone #70443 cherry-pick and is excluded from these counts.
   - **Before and after:** Previously Azure clients only used an account key. 
`TAzureClientConfig` now carries endpoint, account, container and exactly one 
SharedKey/SAS/client-secret credential. The shared SDK client handles each 
method. OAuth tokens refresh inside the SDK; FE supplies replacement SAS 
credentials. Typed configuration takes precedence over legacy properties. 
Native abfs/wasb/HTTPS locations are resolved and checked against the 
configured account and container.
   - **Configs and defaults:** Add positive, mutable 
`azure_client_cache_capacity` (default 256). Azure clients use one bounded LRU 
across containers, with expired entries removed first and SAS expiry checked 
before creation, lookup and publication. Shrinking the limit evicts immediately 
without invalidating existing readers. Other providers retain their existing 
cache behavior.
   - **Impact on existing users:** Existing SharedKey property inputs remain 
supported. Eight new wire fields are optional, and their IDs were checked 
against master and branch-4.2/4.1/4.0/3.1. SAS/OAuth2 producers must wait until 
all participating BEs support the typed field; older BEs ignore it. OAuth 
requires HTTPS. Error text omits signed URLs, client secrets and raw Azure SDK 
response text; HTTP status/request IDs and TLS diagnostics are retained.
   - **Integration:** Scan/schema/metadata readers, Iceberg data/delete/MERGE 
writers, deletion vectors, Hive, TVF, OUTFILE and its success marker propagate 
the typed contract. See `common/cpp/obj-storage/README.md` for fields, entry 
points and rollout guidance. Recycler remains on its existing SharedKey 
metadata path.
   - **Outbound requests:** OAuth introduces token requests to 
`authority_host`, and Blob requests use `endpoint`. These fields arrive through 
trusted FE RPCs; this PR adds no SQL-facing property. Future catalog producers 
must enforce their CREATE/ALTER catalog privileges and endpoint policy before 
emitting them. A catalog creator must not gain unrestricted outbound access 
through this interface.
   
   ### Testing
   
   Per the author's latest instruction, further compilation and tests were 
skipped before submission. No full BE build or live Azure/Iceberg test 
completed.
   
   Before that instruction:
   - Six SDK-independent Azure authentication unit tests passed.
   - Syntax checks passed for the new Azure URI/parser/config tests, shared 
configuration, holder/cache test sources, filesystem adapter and typed-access 
test at their intermediate revisions. These are not a final-tree build claim.
   - Thrift generation succeeded; all eight optional field IDs were audited 
across the five branches above.
   - The prescribed BE UT script stopped during local JDK environment 
validation before executing tests.
   
   Final-tree checks: clang-format 16 through the repository formatting driver, 
`git diff --cached --check`, and independent static review of propagation, 
compatibility, expiry and cache locking. Added tests cover malformed/ambiguous 
credentials, expiry/calendar boundaries, secret redaction, native paths, 
configuration propagation, cache LRU/expiry/concurrency/shrink, and SDK 
credential construction; remaining tests are unexecuted locally.
   
   ### Notes for Reviewers
   
   This PR deliberately has two commits: (1) cherry-pick #70443; (2) only this 
Azure authentication implementation. Review the second commit for this change. 
After #70443 merges, the prerequisite commit can be dropped by rebasing onto 
master, leaving this implementation as the single change.
   
   ### Release note
   
   Add typed BE Azure SAS and OAuth2 client-secret authentication for data-lake 
integration, with a mutable Azure client-cache limit of 256 by default. FE 
catalog configuration and credential vending remain follow-up integration work.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (typed Azure access and bounded Azure cache; see 
Details).
   - Does this need documentation: Interface documentation included; 
user-facing catalog documentation accompanies FE integration.
   - Pre-submission status: Partially verified: further compilation/tests 
skipped at the author's request; full BE build and live Azure integration 
remain unverified.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex
   - Generated: Implementation, tests, interface documentation and this 
description
   - Human review: Pending
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to