whua3 opened a new pull request, #11713:
URL: https://github.com/apache/gravitino/pull/11713
### What changes were proposed in this pull request?
Add Tencent Cloud Object Storage (COS) support for the fileset catalog. The
layout follows the existing OSS / S3 / GCS / Azure modules.
- `bundles/tencent`: source module containing
- `COSFileSystemProvider`: builds a Hadoop `CosNFileSystem` for the
`cosn://` scheme.
- `COSCredentialsProvider`: Hadoop-side provider that bridges
Gravitino-vended `COSSecretKey` credentials to `hadoop-cos`.
- `COSSecretKeyProvider`: server-side `CredentialProvider` that returns
long-lived COS Secret Key credentials.
- `COSUtils`: helper that maps Gravitino properties to Hadoop
`Configuration`.
- `bundles/tencent-bundle`: shaded fat-jar
(`gravitino-tencent-bundle-<version>.jar`) that bundles `hadoop-cos` and
`cos_api-bundle` with relocated packages.
- `api`: new `COSSecretKeyCredential` type.
- `catalogs/catalog-common`: `COSProperties` and `COSCredentialConfig`.
- `catalogs/catalog-fileset`: wire `cosn://` into
`FilesetCatalogPropertiesMetadata`.
- `catalogs/hadoop-common`: register the `cosn` scheme.
- `clients/filesystem-hadoop3-runtime`: include `bundles/tencent` in the
GVFS runtime jar.
- `docs`: new `fileset-catalog-with-cos.md` and a cross-link from
`fileset-catalog.md`.
- Tests: `TestCOSFileSystemProvider`, `TestCOSCredentialProvider`, and
`FilesetCOSCatalogIT`. The IT is skipped on CI when no real COS credentials are
provided, the same way `FilesetOSSCatalogIT` is.
### Why are the changes needed?
#6490 tracks adding Tencent COS support. Without it, users on Tencent Cloud
have to either rely on S3-compatible workarounds, which do not work with
Gravitino's credential vending, or maintain their own fork.
This PR is the first of a chain of PRs against `main`:
- PR-A (this PR): Java side, with the static `cos-secret-key` credential.
- PR-B: server-side STS credential vending (`cos-token`).
- PR-C: Python GVFS support (`COSStorageHandler` and Python-side
`COSSecretKeyCredential`).
- PR-D: Python `COSTokenCredential` wiring. Can be folded into PR-C if
reviewers prefer fewer PRs.
Part of #6490
### Does this PR introduce _any_ user-facing change?
Yes, additive only. Existing fileset / catalog behavior is unchanged. New
surface:
- New fileset location scheme: `cosn://<bucket>/<path>`.
- New fileset catalog properties: `cos-region`, `cos-endpoint` (optional,
defaults to `cos.${region}.myqcloud.com`), `cos-access-key-id`,
`cos-secret-access-key`.
- New credential provider value: `credential-providers=cos-secret-key`
(static AK/SK; STS will follow in PR-B).
- New deployable artifact: `gravitino-tencent-bundle-<version>.jar`.
- New documentation page: [Fileset Catalog with
COS](docs/fileset-catalog-with-cos.md).
### How was this patch tested?
- Unit tests: `:bundles:tencent:test`, `:catalogs:catalog-common:test`,
`:catalogs:catalog-fileset:test`, `:api:test` all pass locally.
- Build: `:bundles:tencent-bundle:shadowJar`,
`:clients:filesystem-hadoop3-runtime:shadowJar`, and `assemble` for all touched
modules pass locally.
- Spotless and Checkstyle pass.
- `FilesetCOSCatalogIT` passes locally against a real COS bucket in
`ap-guangzhou`. It is skipped on CI when no credentials are provided.
- Manual end-to-end verification on a local Gravitino server with the new
bundle jars: created a fileset catalog backed by `cosn://`, then exercised
`hadoop fs -ls`, `-put`, `-get`, `-cat`, `-rm` over `gvfs://fileset/...`, plus
schema/fileset CRUD via the REST API and direct `cosn://...` access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
