XuQianJin-Stars opened a new pull request, #2836: URL: https://github.com/apache/fluss/pull/2836
### Purpose Linked issue: close #2686 This PR adds support for Tencent Cloud Object Storage (COS) as a remote filesystem for Fluss. COS is one of the most widely used cloud storage services in China and this integration enables users to store Fluss remote data on COS, similar to the existing OSS, OBS, S3, and GCS filesystem integrations. ### Brief change log - **`fluss-fs-cos/pom.xml`**: Add module configuration with dependencies on `hadoop-cos` (3.3.5), `cos_api` (5.6.139), and `fluss-fs-hadoop` for Hadoop filesystem abstraction. - **`COSFileSystem`**: Extends `HadoopFileSystem` to wrap Hadoop's `CosNFileSystem`, with lazy-initialized security token provider support for obtaining temporary credentials. - **`COSFileSystemPlugin`**: Implements `FileSystemPlugin` SPI for the `cosn` scheme. Handles Hadoop configuration translation from Fluss config (prefix `fs.cosn.*`), supports three credential modes: static secret key, custom credentials provider, and dynamic security token via `COSSecurityTokenReceiver`. - **`COSSecurityTokenProvider`**: Generates temporary security tokens using Tencent Cloud STS (Security Token Service) based on existing `secretId`/`secretKey` configuration. - **`COSSecurityTokenReceiver`**: Implements `SecurityTokenReceiver` SPI to receive and apply security tokens, configuring `DynamicTemporaryCOSCredentialsProvider` as the Hadoop credentials provider. - **`DynamicTemporaryCOSCredentialsProvider`**: Implements COS SDK's `COSCredentialsProvider` interface, providing `BasicSessionCredentials` from tokens received via `COSSecurityTokenReceiver`. - **`fluss-filesystems/pom.xml`**: Register `fluss-fs-cos` as a sub-module. ### Tests - `COSFileSystemBehaviorITCase`: Integration test for basic COS filesystem behavior (create, read, write, delete) using static credentials (`secretId`/`secretKey`). - `COSWithTokenFileSystemBehaviorITCase`: Integration test for COS filesystem behavior using dynamic security tokens obtained via STS. - `COSWithTokenFileSystemBehaviorBaseITCase`: Base class for token-based filesystem tests, handling filesystem initialization with both static credentials and security tokens. - `COSTestCredentials`: Test utility that reads COS credentials and endpoint from environment variables (`COSN_SECRET_ID`, `COSN_SECRET_KEY`, `COSN_ENDPOINT`, `COSN_BUCKET`). > Note: These are IT (integration tests) that require actual COS credentials and bucket access. They are skipped automatically when credentials are not available via `Assumptions.assumeTrue`. ### API and Format No existing API or storage format changes. This PR only adds a new filesystem plugin that registers via SPI (`META-INF/services/org.apache.fluss.fs.FileSystemPlugin`). New configuration keys introduced (all following standard Hadoop COS conventions): - `fs.cosn.endpoint` — COS region endpoint (e.g., `ap-guangzhou`) - `fs.cosn.userinfo.secretId` — Tencent Cloud secret ID - `fs.cosn.userinfo.secretKey` — Tencent Cloud secret key - `fs.cosn.credentials.provider` — Custom credentials provider class ### Documentation This PR introduces a new feature. Documentation for COS filesystem configuration should be added to the Fluss documentation site in a follow-up. Usage example: ```yaml remote.data.dir: cosn://bucket-name/fluss-data fs.cosn.endpoint: ap-guangzhou fs.cosn.userinfo.secretId: <your-secret-id> fs.cosn.userinfo.secretKey: <your-secret-key> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
