yuqi1129 commented on code in PR #12434:
URL: https://github.com/apache/gravitino/pull/12434#discussion_r3793540258
##########
docs/fileset-catalog-with-cos.md:
##########
@@ -199,346 +196,367 @@ curl -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
"properties": {
"k1": "v1"
}
-}'
http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets
+}'
http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog/schemas/cos_schema/filesets
```
</TabItem>
<TabItem value="java" label="Java">
```java
-GravitinoClient gravitinoClient = GravitinoClient
- .builder("http://localhost:8090";)
- .withMetalake("metalake")
- .build();
-
-Catalog catalog = gravitinoClient.loadCatalog("test_catalog");
+Catalog catalog = gravitinoClient.loadCatalog("cos_catalog");
FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();
-Map<String, String> propertiesMap = ImmutableMap.<String, String>builder()
- .put("k1", "v1")
- .build();
+Map<String, String> filesetProperties = ImmutableMap.<String, String>builder()
+ .put("k1", "v1")
+ .build();
filesetCatalog.createFileset(
- NameIdentifier.of("test_schema", "example_fileset"),
+ NameIdentifier.of("cos_schema", "example_fileset"),
"This is an example fileset",
Fileset.Type.MANAGED,
"cosn://my-bucket-1250000000/root/schema/example_fileset",
- propertiesMap);
+ filesetProperties);
```
</TabItem>
<TabItem value="python" label="Python">
```python
-gravitino_client: GravitinoClient =
GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake")
-
-catalog: Catalog = gravitino_client.load_catalog(name="test_catalog")
-catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema",
"example_fileset"),
- type=Fileset.Type.MANAGED,
- comment="This is an example
fileset",
-
storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset",
- properties={"k1": "v1"})
+catalog: Catalog = gravitino_client.load_catalog(name="cos_catalog")
+catalog.as_fileset_catalog().create_fileset(
+ ident=NameIdentifier.of("cos_schema", "example_fileset"),
+ type=Fileset.Type.MANAGED,
+ comment="This is an example fileset",
+ storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset",
+ properties={"k1": "v1"})
```
</TabItem>
</Tabs>
-## Access a Fileset with COS
+The fileset is now addressable as
+`gvfs://fileset/cos_catalog/cos_schema/example_fileset` from any GVFS client.
+
+## Access the Fileset
-### Access the Fileset with the GVFS Java Client
+### Client jars
-To access fileset with COS using the GVFS Java client, based on the [basic
GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the
following configurations:
+Every client needs `gravitino-filesystem-hadoop3-runtime`, which is published
on Maven Central,
+plus the Tencent Cloud COS filesystem implementation. Only the latter differs
by environment:
+
+| Environment | Jar providing the Tencent Cloud COS filesystem
|
+|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| No Hadoop installed |
[`gravitino-tencent-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle),
a fat jar bundling `hadoop-cos` and the Tencent Cloud COS Java SDK
|
+| Hadoop already present | `hadoop-cos-3.3.0-8.3.23.jar` and
`cos_api-bundle-5.6.227.jar`, published by Tencent Cloud on Maven Central and,
unlike `hadoop-aws` or `hadoop-aliyun`, not part of the Apache Hadoop
distribution |
+
+The artifacts in full:
+
+-
[`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle):
+ a "fat" jar that includes the `gravitino-tencent` functionality together
with every dependency it needs,
+ such as `hadoop-cos` and the Tencent Cloud COS Java SDK. Use it when the
environment has no pre-existing Hadoop setup.
+-
[`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime):
+ a "fat" jar that bundles the Gravitino virtual filesystem client and already
includes the
+ `gravitino-tencent` functionality. It is required for accessing Gravitino
filesets in every environment.
+- `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar`: the standard
Hadoop dependencies
+ for Tencent Cloud COS access, published by Tencent Cloud on Maven Central
and, unlike `hadoop-aws`
+ or `hadoop-aliyun`, not part of the Apache Hadoop distribution. Supply them
yourself when running
+ inside an existing Hadoop environment.
+
+```xml
+<!-- No Hadoop environment -->
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-tencent-bundle</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+```
-| Configuration item | Description
| Default
value | Required |
-|-------------------------|--------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `cos-region` | The region of the Tencent Cloud COS bucket.
| (none)
| Yes |
-| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud COS
service (e.g. `cos.ap-guangzhou.myqcloud.com`, not a full URL). Optional. |
(none) | No |
-| `cos-access-key-id` | The access key ID (Tencent Cloud SecretId) for COS
data. | (none)
| Yes |
-| `cos-secret-access-key` | The secret access key (Tencent Cloud SecretKey)
for COS data. |
(none) | Yes |
+```xml
+<!-- Existing Hadoop environment -->
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${HADOOP_VERSION}</version>
+</dependency>
+<!-- hadoop-cos is published by Tencent Cloud, not by Apache Hadoop. -->
+<dependency>
+ <groupId>com.qcloud.cos</groupId>
+ <artifactId>hadoop-cos</artifactId>
+ <version>3.3.0-8.3.23</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+```
:::note
-If the catalog has enabled [credential
vending](security/credential-vending.md), the AK/SK properties above can be
omitted. More details can be found in [Fileset with credential
vending](#fileset-with-credential-vending).
+The thin `gravitino-tencent` jar is not needed. Its functionality is already
included in both
+`gravitino-tencent-bundle` and `gravitino-filesystem-hadoop3-runtime`.
:::
+### GVFS Java client
+
+On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration),
set the Tencent Cloud COS
+properties from the table above.
+
```java
Configuration conf = new Configuration();
conf.set("fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs");
conf.set("fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
conf.set("fs.gravitino.server.uri", "http://localhost:8090";);
-conf.set("fs.gravitino.client.metalake", "test_metalake");
+conf.set("fs.gravitino.client.metalake", "metalake");
conf.set("cos-region", "ap-guangzhou");
+conf.set("cos-endpoint", "cos.ap-guangzhou.myqcloud.com");
conf.set("cos-access-key-id", "access_key");
conf.set("cos-secret-access-key", "secret_key");
-Path filesetPath = new
Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir");
+
+Path filesetPath = new
Path("gvfs://fileset/cos_catalog/cos_schema/example_fileset/new_dir");
FileSystem fs = filesetPath.getFileSystem(conf);
fs.mkdirs(filesetPath);
-...
-```
-
-Similar to Spark configurations, you need to add COS (bundle) jars to the
classpath according to your environment.
-If you want to customise your hadoop version or there is already a hadoop
version in your project, you can add the following dependencies to your
`pom.xml`:
-
-```xml
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${HADOOP_VERSION}</version>
- </dependency>
-
- <!-- hadoop-cos is published by Tencent Cloud, not by Apache Hadoop. -->
- <dependency>
- <groupId>com.qcloud.cos</groupId>
- <artifactId>hadoop-cos</artifactId>
- <version>3.3.0-8.3.23</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
-```
-
-:::note
-Unlike the S3, OSS, GCS and Azure connectors, COS does **not** ship with
Apache Hadoop. The HCFS adapter for COS is published by Tencent Cloud as
`com.qcloud.cos:hadoop-cos`. Make sure the version you pick is compatible with
your Hadoop version (the `<hadoop>-<sdk>` form encodes both, e.g.
`3.3.0-8.3.23` targets Hadoop 3.3.0).
-:::
-
-Or use the bundle jar with Hadoop environment if there is no Hadoop
environment:
-
-```xml
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-tencent-bundle</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
```
-### Access the Fileset with Spark
+### Apache Spark
-The following code snippet shows how to use **PySpark 3.5.0 with Hadoop
environment(Hadoop 3.3.4)** to access the fileset:
-
-Before running the following code, you need to install required packages:
+The example below uses PySpark 3.5.0 in an environment that already has Hadoop
3.3.4.
```bash
pip install pyspark==3.5.0
pip install apache-gravitino==${GRAVITINO_VERSION}
```
-Then you can run the following code:
```python
-from pyspark.sql import SparkSession
import os
+from pyspark.sql import SparkSession
-gravitino_url = "http://localhost:8090";
-metalake_name = "test"
-
-catalog_name = "your_cos_catalog"
-schema_name = "your_cos_schema"
-fileset_name = "your_cos_fileset"
-
-# JDK8 as follows. JDK17 will be slightly different, you need to add
-# '--conf
"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
-# --conf
"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"'
-# to the submit args.
+# On JDK 17, also add:
+# --conf
"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
+# --conf
"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
os.environ["PYSPARK_SUBMIT_ARGS"] = (
- "--jars "
- "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar,"
+ "--jars
/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar,"
"/path/to/hadoop-cos-3.3.0-8.3.23.jar,"
"/path/to/cos_api-bundle-5.6.227.jar "
"--master local[1] pyspark-shell"
)
-spark = SparkSession.builder \
- .appName("cos_fileset_test") \
- .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs") \
- .config("spark.hadoop.fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") \
- .config("spark.hadoop.fs.gravitino.server.uri", gravitino_url) \
- .config("spark.hadoop.fs.gravitino.client.metalake", "test") \
- .config("spark.hadoop.cos-region", "ap-guangzhou") \
- .config("spark.hadoop.cos-access-key-id", os.environ["COS_ACCESS_KEY_ID"])
\
- .config("spark.hadoop.cos-secret-access-key",
os.environ["COS_SECRET_ACCESS_KEY"]) \
- .config("spark.driver.memory", "2g") \
- .config("spark.driver.port", "2048") \
- .getOrCreate()
+
+spark = (SparkSession.builder
+ .appName("cos_fileset")
+ .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs")
+ .config("spark.hadoop.fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem")
+ .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090";)
+ .config("spark.hadoop.fs.gravitino.client.metalake", "metalake")
+ .config("spark.hadoop.cos-region", "ap-guangzhou")
+ .config("spark.hadoop.cos-endpoint", "cos.ap-guangzhou.myqcloud.com")
+ .config("spark.hadoop.cos-access-key-id", "access_key")
+ .config("spark.hadoop.cos-secret-access-key", "secret_key")
+ .config("spark.driver.memory", "2g")
+ .config("spark.driver.port", "2048")
+ .getOrCreate())
data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)]
-columns = ["Name", "Age"]
-spark_df = spark.createDataFrame(data, schema=columns)
-gvfs_path =
f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people"
-
-spark_df.coalesce(1).write \
- .mode("overwrite") \
- .option("header", "true") \
- .csv(gvfs_path)
+spark_df = spark.createDataFrame(data, schema=["Name", "Age"])
+gvfs_path = "gvfs://fileset/cos_catalog/cos_schema/example_fileset/people"
+
+spark_df.coalesce(1).write.mode("overwrite").option("header",
"true").csv(gvfs_path)
```
-If your Spark is **without Hadoop environment**, you can use the following
code snippet to access the fileset:
+If Spark runs without a Hadoop environment, only the jar list changes:
```python
-## Only the PYSPARK_SUBMIT_ARGS line below changes; keep all the
SparkSession.builder.config(...)
-## calls (including spark.hadoop.cos-region / cos-access-key-id /
cos-secret-access-key) above as-is.
-
-os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars
/path/to/gravitino-tencent-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar
--master local[1] pyspark-shell"
+os.environ["PYSPARK_SUBMIT_ARGS"] = (
+ "--jars /path/to/gravitino-tencent-bundle-${gravitino-version}.jar,"
+ "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar "
+ "--master local[1] pyspark-shell"
+)
```
--
[`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle):
A "fat" JAR that includes `gravitino-tencent` functionality and all necessary
dependencies like `hadoop-cos` and the Tencent Cloud COS Java SDK. Use this if
your Spark environment doesn't have a pre-existing Hadoop setup.
--
[`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime):
A "fat" JAR that bundles Gravitino's virtual filesystem client and includes
the functionality of `gravitino-tencent`. It is required for accessing
Gravitino filesets.
-
-Please choose the correct jar according to your environment.
-
:::note
-In some Spark versions, a Hadoop environment is needed by the driver, adding
the bundle jars with `--jars` may not work. If this is the case, you should add
the jars to the spark CLASSPATH directly.
+Some Spark versions need a Hadoop environment in the driver and do not pick up
filesystem
+implementations passed with `--jars`. If that happens, add the jars to the
Spark classpath directly.
:::
-### Access a Fileset Using the Hadoop Fs Command
-
-The following are examples of how to use the `hadoop fs` command to access the
fileset in Hadoop 3.1.3:
+### Hadoop fs command
-1. Add the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml`
file:
+1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`:
```xml
- <property>
- <name>fs.AbstractFileSystem.gvfs.impl</name>
- <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
- </property>
-
- <property>
- <name>fs.gvfs.impl</name>
-
<value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
- </property>
-
- <property>
- <name>fs.gravitino.server.uri</name>
- <value>http://localhost:8090</value>
- </property>
-
- <property>
- <name>fs.gravitino.client.metalake</name>
- <value>test</value>
- </property>
-
- <property>
- <name>cos-region</name>
- <value>ap-guangzhou</value>
- </property>
-
- <property>
- <name>cos-access-key-id</name>
- <value>access-key</value>
- </property>
-
- <property>
- <name>cos-secret-access-key</name>
- <value>secret-key</value>
- </property>
+<property>
+ <name>fs.AbstractFileSystem.gvfs.impl</name>
+ <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
+</property>
+<property>
+ <name>fs.gvfs.impl</name>
+
<value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
+</property>
+<property>
+ <name>fs.gravitino.server.uri</name>
+ <value>http://localhost:8090</value>
+</property>
+<property>
+ <name>fs.gravitino.client.metalake</name>
+ <value>metalake</value>
+</property>
+<property>
+ <name>cos-region</name>
+ <value>ap-guangzhou</value>
+</property>
+<property>
+ <name>cos-endpoint</name>
+ <value>cos.ap-guangzhou.myqcloud.com</value>
+</property>
+<property>
+ <name>cos-access-key-id</name>
+ <value>access_key</value>
+</property>
+<property>
+ <name>cos-secret-access-key</name>
+ <value>secret_key</value>
+</property>
```
-2. Add the necessary jars to the Hadoop classpath.
+2. Add these jars to the Hadoop classpath:
-For COS, you need to add
`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`,
`hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar` to the Hadoop
classpath. Unlike `hadoop-aws` or `hadoop-aliyun`, these jars are *not* part of
the Apache Hadoop distribution; download them from Maven Central and place them
under `${HADOOP_HOME}/share/hadoop/tools/lib/`.
+ - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from
Maven Central.
+ - `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar`, published
by Tencent Cloud on Maven Central and, unlike `hadoop-aws` or `hadoop-aliyun`,
not part of the Apache Hadoop distribution.
-3. Run the following command to access the fileset:
+3. Access the fileset:
```shell
-./${HADOOP_HOME}/bin/hadoop fs -ls
gvfs://fileset/cos_catalog/cos_schema/cos_fileset
-./${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/cos_catalog/cos_schema/cos_fileset
+${HADOOP_HOME}/bin/hadoop fs -ls
gvfs://fileset/cos_catalog/cos_schema/example_fileset
+${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/cos_catalog/cos_schema/example_fileset
```
-### Access the Fileset with the GVFS Python Client / Pandas
+### GVFS Python client
-:::note
-The GVFS **Python** client does not yet ship a COS storage handler, so reading
-and writing COS-backed filesets via the Python
`gvfs.GravitinoVirtualFileSystem`
-or pandas `read_csv("gvfs://...")` is **not supported in this release**. Only
-the GVFS **Java** client (and Spark / `hadoop fs` on top of it) can do fileset
-I/O against COS today.
-
-The Python `GravitinoClient` itself is not affected: you can still create,
-inspect, update and delete COS catalogs, schemas and filesets through the
-metadata API exactly as shown in the [Step 1 example
above](#step-1-create-a-fileset-catalog-with-cos) and the following Step 2 /
Step 3 sections.
-What is missing is purely the *data plane* (`gvfs.ls`, `gvfs.open`,
-pandas `read_csv("gvfs://...")`, etc.) for COS, which will be added in a
-follow-up release together with a `COSStorageHandler` and the corresponding
-fsspec adapter.
-:::
+```bash
+pip install apache-gravitino==${GRAVITINO_VERSION}
+```
+
+On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Tencent Cloud COS
+properties in `options`, spelled with underscores.
+
+```python
+from gravitino import gvfs
+
+options = {
+ "cache_size": 20,
+ "cache_expired_time": 3600,
+ "auth_type": "simple",
+ "cos_region": "ap-guangzhou",
+ "cos_endpoint": "cos.ap-guangzhou.myqcloud.com",
+ "cos_access_key_id": "access_key",
+ "cos_secret_access_key": "secret_key",
+}
+
+fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090";,
+ metalake_name="metalake",
+ options=options)
+fs.ls("gvfs://fileset/cos_catalog/cos_schema/example_fileset/")
+```
+
+### pandas
+
+pandas reaches the same paths through `storage_options`.
-For other use cases of the Java GVFS client, refer to the [Gravitino Virtual
File System](./how-to-use-gvfs.md) document.
+```python
+import pandas as pd
+
+storage_options = {
+ "server_uri": "http://localhost:8090";,
+ "metalake_name": "metalake",
+ "options": {
+ "cos_region": "ap-guangzhou",
+ "cos_endpoint": "cos.ap-guangzhou.myqcloud.com",
+ "cos_access_key_id": "access_key",
+ "cos_secret_access_key": "secret_key",
+ }
+}
+
+ds =
pd.read_csv("gvfs://fileset/cos_catalog/cos_schema/example_fileset/people/part-00000.csv",
+ storage_options=storage_options)
+ds.head()
+```
-## Fileset with Credential Vending
+For further use cases, see [Gravitino Virtual File
System](./how-to-use-gvfs.md).
-Gravitino supports credential vending for the COS fileset. If the catalog has
been [configured with credential](./security/credential-vending.md), you can
access COS fileset without providing authentication information like
`cos-access-key-id` and `cos-secret-access-key` in the GVFS client
configuration.
+## Credential Vending
-The currently supported credential providers are listed below:
+With credential vending the catalog holds the Tencent Cloud COS credentials
and the Gravitino server hands
+out a credential per request, so clients never hold cloud keys of their own.
See
+[Credential Vending](./security/credential-vending.md) for the general
mechanism and
+[COS credentials](./security/credential-vending.md#) for the properties
Review Comment:
Removed the dead link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
