sollhui opened a new pull request, #70423: URL: https://github.com/apache/doris/pull/70423
### Background Doris BE and Cloud Recycler access Alibaba Cloud OSS through the AWS S3-compatible client. Both already use the shared C++ object storage interface. The FE OSS filesystem plugin uses the Alibaba Cloud Java SDK. ### What problem does this PR solve? Issue Number: #65793 (related native OSS work) Related PR: #65794 and #65795 (existing OSS dependency and filesystem proposals) PR Type: Feature Add Alibaba Cloud OSS C++ SDK V2 0.2.0 as an opt-in object storage client for BE and Recycler. Keep the OSS provider identity across FE property maps and Recycler vault conversion so each process can select its native client. ### Details - **Change map:** The shared adapter implements native requests in about 380 lines of C++. The remaining production changes cover third-party build and license metadata, provider selection and a startup setting. Tests cover provider propagation and the adapter's wire behavior. - **Scope:** Range reads, writes, multipart operations, listing, deletion, GET presigning, versioning and lifecycle inspection. Credential acquisition uses the existing Doris provider and resolves credentials at signing time. This patch does not introduce new Alibaba RAM or OIDC credential modes. - **Design:** BE and Recycler share one adapter. It streams ranged GET responses into the caller's buffer and reads upload bodies from the caller's memory. Recycler sends GetBucketLifecycle through the SDK's generic operation API and extracts NoncurrentDays. The existing outer rate limiter stays in the construction path. - **Usage:** Set `enable_native_oss_sdk=true` in the startup configuration of each BE or Recycler where native OSS access is wanted, then restart that process. - **Default behavior:** The setting defaults to false. Existing installations keep their S3-compatible request path. - **Compatibility:** The setting applies to identified OSS storage. AWS and the other providers retain their existing clients. The existing FE Java OSS SDK dependency is unchanged. ### Testing - Built the pinned upstream 0.2.0 SDK as a static library on macOS arm64. Compiled and linked the new Doris adapter and shared `ObjStorageClient` against that library. - Exercised the compiled adapter against a loopback HTTP service. The fixture checked OSS V4 authorization and the session token on requests. PUT, ranged GET, HEAD, multipart initiate/upload/complete/abort, a two-page list including object timestamps, single and batch delete, versioning, noncurrent-version lifecycle expiration and GET presigning passed. - The new HTTP unit test translation unit passed a syntax-only compile. The BE header hygiene gate, clang-format 16 checks, shell syntax checks and `git diff --check` passed. - Full BE/Recycler build and the repository test binaries have not been verified. The available third-party installation predates the master build requirements. The FE Maven reactor could not resolve the MongoDB BOM (HTTP 403), and a standalone OSS module run could not resolve its local SPI snapshot. No live Alibaba Cloud bucket test or throughput benchmark was run. ### Release note BE and Recycler can use the native Alibaba Cloud OSS C++ SDK V2 for OSS storage when `enable_native_oss_sdk` is enabled. ### Check List (For Author) - Behavior changed: Yes, when native access is enabled (see Details). - Does this need documentation: Yes, document the startup setting in the BE and Recycler configuration reference. - Pre-submission status: Partially verified. Full build, project test binaries and live OSS validation are outstanding. ### AI Disclosure - Tool: OpenAI Codex - Generated: Implementation, tests and PR description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
