sollhui opened a new pull request, #70657: URL: https://github.com/apache/doris/pull/70657
### Background BE and Recycler share provider-specific object-storage configuration and SDK clients after #70443. Azure currently accepts SharedKey through legacy S3-style properties; those properties cannot represent vended SAS or Entra ID client credentials. ### What problem does this PR solve? Issue Number: None Related PR: #70443 (prerequisite, cherry-picked as the first commit), #70386 (typed Azure design reference) PR Type: Enhancement Problem Summary: Give data-lake connectors one typed FE-to-BE Azure access contract supporting SharedKey, SAS and OAuth2 client credentials. Apply it throughout reads and writes, including Iceberg delete files and deletion vectors. FE catalog properties and credential vending will use this interface in a follow-up; this PR implements the protocol and BE/shared storage side. ### Details - **Change map:** The implementation commit adds/removes 1,238/132 production lines across 41 files, 817/9 test lines across 8 files, and 69 documentation lines. Most caller changes only forward the optional configuration. No generated files are committed. The first commit is the standalone #70443 cherry-pick and is excluded from these counts. - **Before and after:** Previously Azure clients only used an account key. `TAzureClientConfig` now carries endpoint, account, container and exactly one SharedKey/SAS/client-secret credential. The shared SDK client handles each method. OAuth tokens refresh inside the SDK; FE supplies replacement SAS credentials. Typed configuration takes precedence over legacy properties. Native abfs/wasb/HTTPS locations are resolved and checked against the configured account and container. - **Configs and defaults:** Add positive, mutable `azure_client_cache_capacity` (default 256). Azure clients use one bounded LRU across containers, with expired entries removed first and SAS expiry checked before creation, lookup and publication. Shrinking the limit evicts immediately without invalidating existing readers. Other providers retain their existing cache behavior. - **Impact on existing users:** Existing SharedKey property inputs remain supported. Eight new wire fields are optional, and their IDs were checked against master and branch-4.2/4.1/4.0/3.1. SAS/OAuth2 producers must wait until all participating BEs support the typed field; older BEs ignore it. OAuth requires HTTPS. Error text omits signed URLs, client secrets and raw Azure SDK response text; HTTP status/request IDs and TLS diagnostics are retained. - **Integration:** Scan/schema/metadata readers, Iceberg data/delete/MERGE writers, deletion vectors, Hive, TVF, OUTFILE and its success marker propagate the typed contract. See `common/cpp/obj-storage/README.md` for fields, entry points and rollout guidance. Recycler remains on its existing SharedKey metadata path. - **Outbound requests:** OAuth introduces token requests to `authority_host`, and Blob requests use `endpoint`. These fields arrive through trusted FE RPCs; this PR adds no SQL-facing property. Future catalog producers must enforce their CREATE/ALTER catalog privileges and endpoint policy before emitting them. A catalog creator must not gain unrestricted outbound access through this interface. ### Testing Per the author's latest instruction, further compilation and tests were skipped before submission. No full BE build or live Azure/Iceberg test completed. Before that instruction: - Six SDK-independent Azure authentication unit tests passed. - Syntax checks passed for the new Azure URI/parser/config tests, shared configuration, holder/cache test sources, filesystem adapter and typed-access test at their intermediate revisions. These are not a final-tree build claim. - Thrift generation succeeded; all eight optional field IDs were audited across the five branches above. - The prescribed BE UT script stopped during local JDK environment validation before executing tests. Final-tree checks: clang-format 16 through the repository formatting driver, `git diff --cached --check`, and independent static review of propagation, compatibility, expiry and cache locking. Added tests cover malformed/ambiguous credentials, expiry/calendar boundaries, secret redaction, native paths, configuration propagation, cache LRU/expiry/concurrency/shrink, and SDK credential construction; remaining tests are unexecuted locally. ### Notes for Reviewers This PR deliberately has two commits: (1) cherry-pick #70443; (2) only this Azure authentication implementation. Review the second commit for this change. After #70443 merges, the prerequisite commit can be dropped by rebasing onto master, leaving this implementation as the single change. ### Release note Add typed BE Azure SAS and OAuth2 client-secret authentication for data-lake integration, with a mutable Azure client-cache limit of 256 by default. FE catalog configuration and credential vending remain follow-up integration work. ### Check List (For Author) - Behavior changed: Yes (typed Azure access and bounded Azure cache; see Details). - Does this need documentation: Interface documentation included; user-facing catalog documentation accompanies FE integration. - Pre-submission status: Partially verified: further compilation/tests skipped at the author's request; full BE build and live Azure integration remain unverified. ### AI Disclosure - Tool: OpenAI Codex - Generated: Implementation, tests, interface documentation and this description - Human review: Pending -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
