sollhui opened a new pull request, #70443: URL: https://github.com/apache/doris/pull/70443
### Background BE creates S3-compatible and Azure Blob clients from FE properties, storage resource parameters, and vault metadata. The former S3 utility owned input conversion, SDK setup, client caching and provider construction. File systems also keep a holder so a credential refresh can replace the client used by existing readers and writers. ### What problem does this PR solve? Issue Number: None Related PR: #69605 (alternative design for the BE client module) PR Type: Refactor Separate configuration, client reuse, SDK construction and live client replacement. Both backends continue to use one cache keyed by the complete input configuration, including fields that the selected SDK does not consume. ### Behavior changes and bug fixes for separate PRs These items are excluded from this patch. 1. Make it safe to read AWS client defaults concurrently. The existing accessor writes the timeout into the same static configuration on every call. 2. Release cached AWS clients before SDK shutdown. The current factory shuts down the SDK before its cache members are destroyed. 3. Reject trailing characters in integer properties. The current parser accepts the numeric prefix of a value such as `12x`. 4. Give callers a synchronized configuration snapshot when reading during a holder reset. The existing configuration accessor returns a reference to mutable state, and path formatting also reads the configuration without taking the reset lock. ### Details - **Change map:** Production code +1482/-1134, tests +279/-235. Most of the implementation comes from the old S3 utility and file system. The BE callers and test fixtures use the new names and headers. - **Moves vs. logic changes:** Under `be/src/io/object_storage/`, the config parser handles the three input formats. The config value owns validation, identity and refresh field selection. The factory handles cache lookup, backend dispatch, rate-limit wrapping and publication. Separate S3 and Azure builders own SDK-specific construction. AWS settings are shared with Kinesis. The holder owns the active client and its refresh. The Azure auth API in common and the cloud Recycler are outside this change. - **Behavior change:** None intended. A refresh retains the previous endpoint, region and endpoint-override flag. The other client settings come from the update. Construction happens outside the cache lock. Concurrent misses publish the first inserted client, and a failed refresh leaves the active client in place. - **Compatibility:** FE properties, metadata, validation order, cache equality, SDK options and refresh entry points keep their existing semantics. Azure continues to use SharedKey. - **Performance impact:** Cache lookup and locking follow the existing implementation. No performance benchmark was run. - **Rollback:** Revert the commit. No configuration or metadata migration is involved. ### Testing - Local clang 16 syntax checks passed for all eight new production translation units with Azure enabled (BE_TEST) and disabled. - The new cross-backend refresh GoogleTest was compiled and run separately and passed. It checks which fields move with the update in both directions. - The BE build-hygiene checks and staged diff whitespace check passed. - A full local BE build stopped during third-party setup because LLVM 20 is missing. The full BE unit suite and service integration tests were not run. ### Release note None. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified. Full BE build and test suite pending. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Refactor, test adaptation, refresh test and PR description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
