whua3 commented on code in PR #11713:
URL: https://github.com/apache/gravitino/pull/11713#discussion_r3487753231
##########
docs/fileset-catalog-with-cos.md:
##########
@@ -0,0 +1,548 @@
+---
+title: "Fileset Catalog with COS"
+slug: "/fileset-catalog-with-cos"
+date: 2026-06-17
+keyword: "Fileset catalog COS Tencent"
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Introduction
+
+This document explains how to configure a Fileset catalog with Tencent Cloud
COS (Cloud Object Storage) in Gravitino.
+
+## Prerequisites
+
+To set up a Fileset catalog with COS, follow these steps:
+
+1. Download the
[`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle)
file.
+2. Place the downloaded file into the Gravitino Fileset catalog classpath at
`${GRAVITINO_HOME}/catalogs/fileset/libs/`.
+3. Start the Gravitino server by running the following command:
+
+```bash
+$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start
+```
+
+Once the server is up and running, you can proceed to configure the Fileset
catalog with COS. In the rest of this document we will use
`http://localhost:8090` as the Gravitino server URL, replace with your actual
server URL.
+
+## COS Catalog Configuration
+
+### COS Fileset Catalog Configuration
+
+In addition to the basic configurations mentioned in
[Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties),
the following properties are required to configure a Fileset catalog with COS:
+
+| Configuration item | Description
| Default value |
Required | Since version |
+|-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
+| `cos-region` | The region of the Tencent Cloud COS bucket,
e.g. `ap-guangzhou`, `ap-shanghai`.
| (none)
| Yes | 1.4.0 |
+| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud
COS service (mapped to `fs.cosn.bucket.endpoint_suffix`). It is a host suffix,
not a full URL — e.g. `cos.ap-guangzhou.myqcloud.com`, **not**
`https://cos.ap-guangzhou.myqcloud.com`. Optional; when not set, hadoop-cos
derives it from `cos-region` (`cos.${region}.myqcloud.com`). Set this only if
you need to point to a non-public endpoint (e.g. an internal/VPC endpoint).
| (none) | No | 1.4.0 |
+| `cos-access-key-id` | The static access key ID (Tencent Cloud
`SecretId`) used to access COS data.
| (none)
| Yes | 1.4.0 |
+| `cos-secret-access-key` | The static secret access key (Tencent Cloud
`SecretKey`) used to access COS data.
| (none)
| Yes | 1.4.0 |
+| `credential-providers` | The credential provider types, separated by
comma. The currently supported value is `cos-secret-key`. Setting this enables
credential vending provided by the Gravitino server, so the GVFS client no
longer needs `cos-access-key-id` / `cos-secret-access-key` locally. See
[cos-credential-vending](#fileset-with-credential-vending) below for details. |
(none) | No | 1.4.0 |
+
+:::note
+The fileset catalog automatically loads filesystem providers on the classpath.
The COS provider is registered when the `gravitino-tencent-bundle` jar is
present, so `default-filesystem-provider` and `filesystem-providers` do not
need to be set.
+:::
+
+:::note
+`cos-region` is mandatory for hadoop-cos: signing requests, building the
default endpoint and selecting the right CAM scope all require the region. Even
if you also set `cos-endpoint`, please keep `cos-region` set.
+:::
+
+### Schema Configuration
+
+To create a schema, refer to [Schema
configurations](./fileset-catalog.md#schema-properties).
+
+### Fileset Configuration
+
+For instructions on how to create a fileset, refer to [Fileset
configurations](./fileset-catalog.md#fileset-properties) for more details.
+
+## Create the Catalog, Schema, and Fileset
+
+This section will show you how to use the Fileset catalog with COS in
Gravitino, including detailed examples.
+
+### Step 1: Create a Fileset Catalog with COS
+
+First, you need to create a Fileset catalog for COS. The following examples
demonstrate how to create a Fileset catalog with COS:
+
+<Tabs groupId="language" queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "test_catalog",
+ "type": "FILESET",
+ "comment": "This is a COS fileset catalog",
+ "properties": {
+ "location": "cosn://my-bucket-1250000000/root",
+ "cos-region": "ap-guangzhou",
+ "cos-access-key-id": "access_key",
+ "cos-secret-access-key": "secret_key"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+GravitinoClient gravitinoClient = GravitinoClient
+ .builder("http://localhost:8090";)
+ .withMetalake("metalake")
+ .build();
+
+Map<String, String> cosProperties = ImmutableMap.<String, String>builder()
+ .put("location", "cosn://my-bucket-1250000000/root")
+ .put("cos-region", "ap-guangzhou")
+ .put("cos-access-key-id", "access_key")
+ .put("cos-secret-access-key", "secret_key")
+ .build();
+
+Catalog cosCatalog = gravitinoClient.createCatalog("test_catalog",
+ Type.FILESET,
+ "This is a COS fileset catalog",
+ cosProperties);
+// ...
+
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python
+gravitino_client: GravitinoClient =
GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake")
+cos_properties = {
+ "location": "cosn://my-bucket-1250000000/root",
+ "cos-region": "ap-guangzhou",
+ "cos-access-key-id": "access_key",
+ "cos-secret-access-key": "secret_key"
+}
+
+cos_catalog = gravitino_client.create_catalog(name="test_catalog",
+
catalog_type=Catalog.Type.FILESET,
+ provider=None,
+ comment="This is a COS fileset
catalog",
+ properties=cos_properties)
+```
+
+</TabItem>
+</Tabs>
+
+:::note
+Tencent Cloud COS bucket names always end with the appid suffix, for example
`my-bucket-1250000000`. Use that full bucket name in `location` and in any
`cosn://` URI.
+:::
+
+### Step 2: Create a Schema
+
+Once the Fileset catalog with COS is created, you can create a schema inside
that catalog. Below are examples of how to do this:
+
+<Tabs groupId="language" queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "test_schema",
+ "comment": "This is a COS schema",
+ "properties": {
+ "location": "cosn://my-bucket-1250000000/root/schema"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+Catalog catalog = gravitinoClient.loadCatalog("test_catalog");
+
+SupportsSchemas supportsSchemas = catalog.asSchemas();
+
+Map<String, String> schemaProperties = ImmutableMap.<String, String>builder()
+ .put("location", "cosn://my-bucket-1250000000/root/schema")
+ .build();
+Schema schema = supportsSchemas.createSchema("test_schema",
+ "This is a COS schema",
+ schemaProperties
+);
+// ...
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python
+gravitino_client: GravitinoClient =
GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake")
+catalog: Catalog = gravitino_client.load_catalog(name="test_catalog")
+catalog.as_schemas().create_schema(name="test_schema",
+ comment="This is a COS schema",
+ properties={"location":
"cosn://my-bucket-1250000000/root/schema"})
+```
+
+</TabItem>
+</Tabs>
+
+
+### Step 3: Create a Fileset
+
+Now that the schema is created, you can create a fileset inside it. Here's how:
+
+<Tabs groupId="language" queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "example_fileset",
+ "comment": "This is an example fileset",
+ "type": "MANAGED",
+ "storageLocation": "cosn://my-bucket-1250000000/root/schema/example_fileset",
+ "properties": {
+ "k1": "v1"
+ }
+}'
http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+GravitinoClient gravitinoClient = GravitinoClient
+ .builder("http://localhost:8090";)
+ .withMetalake("metalake")
+ .build();
+
+Catalog catalog = gravitinoClient.loadCatalog("test_catalog");
+FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();
+
+Map<String, String> propertiesMap = ImmutableMap.<String, String>builder()
+ .put("k1", "v1")
+ .build();
+
+filesetCatalog.createFileset(
+ NameIdentifier.of("test_schema", "example_fileset"),
+ "This is an example fileset",
+ Fileset.Type.MANAGED,
+ "cosn://my-bucket-1250000000/root/schema/example_fileset",
+ propertiesMap,
+);
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python
+gravitino_client: GravitinoClient =
GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake")
+
+catalog: Catalog = gravitino_client.load_catalog(name="test_catalog")
+catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema",
"example_fileset"),
+ type=Fileset.Type.MANAGED,
+ comment="This is an example
fileset",
+
storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset",
+ properties={"k1": "v1"})
+```
+
+</TabItem>
+</Tabs>
+
+## Access a Fileset with COS
+
+### Access the Fileset with the GVFS Java Client
+
+To access fileset with COS using the GVFS Java client, based on the [basic
GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the
following configurations:
+
+| Configuration item | Description
| Default value | Required | Since version |
+|-------------------------|--------------------------------------------------------------|---------------|----------|---------------|
+| `cos-region` | The region of the Tencent Cloud COS bucket.
| (none) | Yes | 1.4.0 |
+| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud COS
service (e.g. `cos.ap-guangzhou.myqcloud.com`, not a full URL). Optional. |
(none) | No | 1.4.0 |
+| `cos-access-key-id` | The access key ID (Tencent Cloud SecretId) for COS
data. | (none) | Yes | 1.4.0 |
+| `cos-secret-access-key` | The secret access key (Tencent Cloud SecretKey)
for COS data.| (none) | Yes | 1.4.0 |
+
+:::note
+If the catalog has enabled [credential
vending](security/credential-vending.md), the AK/SK properties above can be
omitted. More details can be found in [Fileset with credential
vending](#fileset-with-credential-vending).
+:::
+
+```java
+Configuration conf = new Configuration();
+conf.set("fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs");
+conf.set("fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
+conf.set("fs.gravitino.server.uri", "http://localhost:8090";);
+conf.set("fs.gravitino.client.metalake", "test_metalake");
+conf.set("cos-region", "ap-guangzhou");
+conf.set("cos-access-key-id", "access_key");
+conf.set("cos-secret-access-key", "secret_key");
+Path filesetPath = new
Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir");
+FileSystem fs = filesetPath.getFileSystem(conf);
+fs.mkdirs(filesetPath);
+...
+```
+
+Similar to Spark configurations, you need to add COS (bundle) jars to the
classpath according to your environment.
+If you want to customise your hadoop version or there is already a hadoop
version in your project, you can add the following dependencies to your
`pom.xml`:
+
+```xml
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${HADOOP_VERSION}</version>
+ </dependency>
+
+ <!-- hadoop-cos is published by Tencent Cloud, not by Apache Hadoop. -->
+ <dependency>
+ <groupId>com.qcloud.cos</groupId>
+ <artifactId>hadoop-cos</artifactId>
+ <version>3.3.0-8.3.23</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+ </dependency>
+```
+
+:::note
+Unlike the S3, OSS, GCS and Azure connectors, COS does **not** ship with
Apache Hadoop. The HCFS adapter for COS is published by Tencent Cloud as
`com.qcloud.cos:hadoop-cos`. Make sure the version you pick is compatible with
your Hadoop version (the `<hadoop>-<sdk>` form encodes both, e.g.
`3.3.0-8.3.23` targets Hadoop 3.3.0).
+:::
+
+:::note
+Since version 1.4.0, the `gravitino-tencent` JAR is no longer required
separately, as it is included in the `gravitino-filesystem-hadoop3-runtime` JAR.
+:::
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
