xylaaaaa opened a new pull request, #70386: URL: https://github.com/apache/doris/pull/70386
> **Depends on #69667, which depends on #69605.** This PR is stacked on #69667 (BE-2 of the #69082 split), so until #69605 and #69667 are merged its diff also contains their commits. Only the last commit (`[feature](be) Accept typed Azure access with SAS and OAuth2 from FE`) belongs to this PR. It will be rebased onto master after they are merged. ### What problem does this PR solve? Issue Number: None Related PR: #69082, #69605, #69667 Problem Summary: BE can reach Azure Blob Storage only with SharedKey: every entry hands BE the S3-style properties (account and key as ak/sk, the container as the bucket), so a SAS token or an Entra ID client secret has nowhere to go, and a native location such as abfss://[email protected]/path is not understood by BE. Short-lived credentials also need a bound on the Azure client cache: each SAS token is a new client identity, so a catalog that vends a fresh SAS per query would add one client, with its own connection pool, per query. This is BE-3 of the split of #69082 and is based on #69667 (BE-2), which is based on #69605 (BE-1). It only adds the BE side; FE does not send the new fields yet (FE-1), and without them BE takes the same path as before. 1. Protocol. `TAzureClientConfig` (endpoint, account name, container, and a `TAzureCredential` union of SharedKey, SAS with an optional expiry, and client secret) is added as an optional field of `TFileScanRangeParams` (43), `TParquetMetadataParams` (7), `TResultFileSinkOptions` (24), `THiveTableSink` (15), `TIcebergTableSink` (21), `TIcebergDeleteSink` (16), `TIcebergMergeSink` (26) and `TTVFTableSink` (17). None of these ids is used on master, branch-4.2, branch-4.1, branch-4.0 or branch-3.1. 2. Authentication. SAS and client secret are added as new alternatives of the common `AzureAuthConfig` (`common/cpp/obj-client/auth/azure/`), with their required fields, cache key hash, names, and SDK construction in `AzureAuthFactory`. The SAS expiry is the earlier of the upstream expiry and the token's `se` field; an expired SAS, or one whose expiry cannot be read, is rejected before a client is built and again before it enters the cache. Presigned URLs keep using only the SharedKey account key, so a SAS token or a client secret never appears there. Recycler still builds SharedKey only from its storage metadata, which cannot carry the other methods. 3. Decoding. `ObjStorageConfigResolver::from_azure_client_config` turns `TAzureClientConfig` and the file location into a validated `ObjClientConfig`: `AzureConfigCodec` resolves the shape (exactly one credential; a configured container must match the location's), `AzureConfigValidator` the rules (endpoint, container, the required fields of the method, and for a standard Azure endpoint the account it names must be the configured one, so a request is never signed for one account and sent to another). An account named by the location's host must match too, e.g. abfss://[email protected]/k with account "acct" is rejected. 4. Locations. `ObjStorageUri::parse_azure` accepts every Azure form FE can send (abfs[s]://, wasb[s]://, http[s]://acct.blob..., the s3-style form, and a bare key) and records the account the host names. `parse_s3` now also accepts abfs[s]:// and wasb[s]://, taking only container and key as for the s3-style form FE rewrites them to today. 5. Read. `FileSystemProperties` / `FSPropertiesRef` carry the typed config, and `FileFactory::create_fs` and `create_file_writer` build the Azure client from it when it is set, ignoring the S3-style Azure keys. The V2 readers (`TableReader`, schema inference, Iceberg delete files, deletion vectors) and `parquet_meta` copy it from the scan params. 6. Write and other entries. The Iceberg data, delete and merge sinks, the Hive sink, OUTFILE (including `outfile_write_success`) and TVF insert pass the config to their file writers. 7. Cache. Azure clients are cached with a bound, `azure_client_cache_capacity` (default 256, mutable). Expired entries are evicted first, then the least recently used; eviction only drops the cache's reference, so a reader that holds a client keeps it. A lowered capacity applies at once, and a publish cannot undo it with the value it read before. S3 clients stay unbounded. ### Release note Add the BE config `azure_client_cache_capacity` (default 256, mutable), the maximum number of cached native Azure clients. It takes effect once FE sends typed Azure access. ### Check List (For Author) - Test: Unit Test (azure_config_test, azure_typed_access_test, obj_client_cache_test, new cases in obj_client_factory_test, obj_storage_uri_test and async_result_writer_test). Not compiled or run locally; CI compiles and runs them. clang-format 16 and build-support/check-build-hygiene.sh pass. - Behavior changed: Yes. `ObjStorageUri::parse_s3` accepts abfs[s]:// and wasb[s]:// locations, which it rejected before; a new BE config. Nothing else changes until FE sends the new fields. - Does this need documentation: No (the config will be documented with FE-1, which makes it reachable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
