Copilot commented on code in PR #12434:
URL: https://github.com/apache/gravitino/pull/12434#discussion_r3764205271
##########
docs/how-to-use-gvfs.md:
##########
@@ -439,7 +442,7 @@ To configure the Gravitino client, use properties prefixed
with `gvfs_gravitino_
:::note
When users work with a multi-cluster fileset catalog, they can configure
separate sets of properties for the base paths
-of the different clusters. [Manage filesets across multiple
clusters](./manage-fileset-metadata-using-gravitino.md#manage-filesets-across-multiple-clusters)
+of the different clusters, using the `fs.path.config.<name>` properties
described above.
Review Comment:
In the Python GVFS section, this note points users to the Java property
prefix `fs.path.config.<name>`, but the Python configuration keys are
`fs_path_config_<name>` (as listed in the table above). As written, this is
likely to confuse users trying to configure multi-cluster paths from Python.
##########
docs/how-to-use-gvfs.md:
##########
@@ -471,12 +474,16 @@ options = {
The plain `fs_path_config_<name>` entry specifies the base path of the
filesystem. Any additional key under the same prefix
(`fs_path_config_<name>_<config_key>`) is treated as a location-scoped
configuration (for example, `config.resource` for HDFS) and is forwarded
directly to the underlying filesystem client.
:::
-#### Configurations for S3, GCS, OSS and Azure Blob Storage Fileset
+#### Configurations for Cloud Storage Filesets
-Please see the cloud-storage-specific configurations [GCS GVFS Python client
configurations](./fileset-catalog-with-gcs.md#access-the-fileset-with-the-gvfs-python-client),
-[S3 GVFS Python client
configurations](./fileset-catalog-with-s3.md#access-the-fileset-with-the-gvfs-python-client),
-[OSS GVFS Python client
configurations](./fileset-catalog-with-oss.md#access-the-fileset-with-the-gvfs-python-client)
-and [Azure Blob Storage GVFS Python client
configurations](./fileset-catalog-with-adls.md#access-the-fileset-with-the-gvfs-python-client)
for more details.
+A fileset backed by cloud storage needs the credential properties of that
backend, spelled with
+underscores rather than hyphens. See
+[Amazon S3](./fileset-catalog-with-s3.md#amazon-s3-properties),
+[Google Cloud
Storage](./fileset-catalog-with-gcs.md#google-cloud-storage-properties),
+[Azure Data Lake
Storage](./fileset-catalog-with-adls.md#azure-data-lake-storage-properties),
+[Alibaba Cloud
OSS](./fileset-catalog-with-oss.md#alibaba-cloud-oss-properties) and
+[Tencent Cloud
COS](./fileset-catalog-with-cos.md#tencent-cloud-cos-properties) for the
property
+names and a runnable example for each backend.
Review Comment:
This Python GVFS section claims there is a runnable Python example for
Tencent Cloud COS, but the Python GVFS implementation does not support
`cosn://` locations (no COS storage handler/prefix is registered; `StorageType`
in `clients/client-python/gravitino/filesystem/gvfs_storage_handler.py` only
includes file/hdfs/gs/s3a/oss/abfss). This link list should not imply COS works
from Python unless the client gains COS support.
##########
docs/fileset-catalog-with-cos.md:
##########
@@ -199,346 +185,348 @@ curl -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
"properties": {
"k1": "v1"
}
-}'
http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets
+}'
http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog/schemas/cos_schema/filesets
```
</TabItem>
<TabItem value="java" label="Java">
```java
-GravitinoClient gravitinoClient = GravitinoClient
- .builder("http://localhost:8090";)
- .withMetalake("metalake")
- .build();
-
-Catalog catalog = gravitinoClient.loadCatalog("test_catalog");
+Catalog catalog = gravitinoClient.loadCatalog("cos_catalog");
FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();
-Map<String, String> propertiesMap = ImmutableMap.<String, String>builder()
- .put("k1", "v1")
- .build();
+Map<String, String> filesetProperties = ImmutableMap.<String, String>builder()
+ .put("k1", "v1")
+ .build();
filesetCatalog.createFileset(
- NameIdentifier.of("test_schema", "example_fileset"),
+ NameIdentifier.of("cos_schema", "example_fileset"),
"This is an example fileset",
Fileset.Type.MANAGED,
"cosn://my-bucket-1250000000/root/schema/example_fileset",
- propertiesMap);
+ filesetProperties);
```
</TabItem>
<TabItem value="python" label="Python">
```python
-gravitino_client: GravitinoClient =
GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake")
-
-catalog: Catalog = gravitino_client.load_catalog(name="test_catalog")
-catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema",
"example_fileset"),
- type=Fileset.Type.MANAGED,
- comment="This is an example
fileset",
-
storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset",
- properties={"k1": "v1"})
+catalog: Catalog = gravitino_client.load_catalog(name="cos_catalog")
+catalog.as_fileset_catalog().create_fileset(
+ ident=NameIdentifier.of("cos_schema", "example_fileset"),
+ type=Fileset.Type.MANAGED,
+ comment="This is an example fileset",
+ storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset",
+ properties={"k1": "v1"})
```
</TabItem>
</Tabs>
-## Access a Fileset with COS
+The fileset is now addressable as
+`gvfs://fileset/cos_catalog/cos_schema/example_fileset` from any GVFS client.
+
+## Access the Fileset
-### Access the Fileset with the GVFS Java Client
+### Client jars
-To access fileset with COS using the GVFS Java client, based on the [basic
GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the
following configurations:
+Every client needs `gravitino-filesystem-hadoop3-runtime`, plus the Tencent
Cloud COS filesystem
+implementation. Which jar provides it depends on the environment:
-| Configuration item | Description
| Default
value | Required |
-|-------------------------|--------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `cos-region` | The region of the Tencent Cloud COS bucket.
| (none)
| Yes |
-| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud COS
service (e.g. `cos.ap-guangzhou.myqcloud.com`, not a full URL). Optional. |
(none) | No |
-| `cos-access-key-id` | The access key ID (Tencent Cloud SecretId) for COS
data. | (none)
| Yes |
-| `cos-secret-access-key` | The secret access key (Tencent Cloud SecretKey)
for COS data. |
(none) | Yes |
+| Environment | Jars to add |
+|-------------|-------------|
+| No Hadoop installed |
[`gravitino-tencent-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle)
— a fat jar bundling `hadoop-cos` and the Tencent Cloud COS Java SDK |
+| Hadoop already present | `hadoop-cos-3.3.0-8.3.23.jar` and
`cos_api-bundle-5.6.227.jar` — unlike `hadoop-aws` or `hadoop-aliyun` these are
not part of the Apache Hadoop distribution, download them from Maven Central |
+
+```xml
+<!-- No Hadoop environment -->
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-tencent-bundle</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+```
+
+```xml
+<!-- Existing Hadoop environment -->
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${HADOOP_VERSION}</version>
+</dependency>
+<!-- hadoop-cos is published by Tencent Cloud, not by Apache Hadoop. -->
+<dependency>
+ <groupId>com.qcloud.cos</groupId>
+ <artifactId>hadoop-cos</artifactId>
+ <version>3.3.0-8.3.23</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.gravitino</groupId>
+ <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
+ <version>${GRAVITINO_VERSION}</version>
+</dependency>
+```
:::note
-If the catalog has enabled [credential
vending](security/credential-vending.md), the AK/SK properties above can be
omitted. More details can be found in [Fileset with credential
vending](#fileset-with-credential-vending).
+The thin `gravitino-tencent` jar is not needed. Its functionality is already
included in both
+`gravitino-tencent-bundle` and `gravitino-filesystem-hadoop3-runtime`.
:::
+### GVFS Java client
+
+On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration),
set the Tencent Cloud COS
+properties from the table above.
+
```java
Configuration conf = new Configuration();
conf.set("fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs");
conf.set("fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
conf.set("fs.gravitino.server.uri", "http://localhost:8090";);
-conf.set("fs.gravitino.client.metalake", "test_metalake");
+conf.set("fs.gravitino.client.metalake", "metalake");
conf.set("cos-region", "ap-guangzhou");
+conf.set("cos-endpoint", "cos.ap-guangzhou.myqcloud.com");
conf.set("cos-access-key-id", "access_key");
conf.set("cos-secret-access-key", "secret_key");
-Path filesetPath = new
Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir");
+
+Path filesetPath = new
Path("gvfs://fileset/cos_catalog/cos_schema/example_fileset/new_dir");
FileSystem fs = filesetPath.getFileSystem(conf);
fs.mkdirs(filesetPath);
-...
-```
-
-Similar to Spark configurations, you need to add COS (bundle) jars to the
classpath according to your environment.
-If you want to customise your hadoop version or there is already a hadoop
version in your project, you can add the following dependencies to your
`pom.xml`:
-
-```xml
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${HADOOP_VERSION}</version>
- </dependency>
-
- <!-- hadoop-cos is published by Tencent Cloud, not by Apache Hadoop. -->
- <dependency>
- <groupId>com.qcloud.cos</groupId>
- <artifactId>hadoop-cos</artifactId>
- <version>3.3.0-8.3.23</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
-```
-
-:::note
-Unlike the S3, OSS, GCS and Azure connectors, COS does **not** ship with
Apache Hadoop. The HCFS adapter for COS is published by Tencent Cloud as
`com.qcloud.cos:hadoop-cos`. Make sure the version you pick is compatible with
your Hadoop version (the `<hadoop>-<sdk>` form encodes both, e.g.
`3.3.0-8.3.23` targets Hadoop 3.3.0).
-:::
-
-Or use the bundle jar with Hadoop environment if there is no Hadoop
environment:
-
-```xml
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-tencent-bundle</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.gravitino</groupId>
- <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
- <version>${GRAVITINO_VERSION}</version>
- </dependency>
```
-### Access the Fileset with Spark
-
-The following code snippet shows how to use **PySpark 3.5.0 with Hadoop
environment(Hadoop 3.3.4)** to access the fileset:
+### Apache Spark
-Before running the following code, you need to install required packages:
+The example below uses PySpark 3.5.0 in an environment that already has Hadoop
3.3.4.
```bash
pip install pyspark==3.5.0
pip install apache-gravitino==${GRAVITINO_VERSION}
```
-Then you can run the following code:
```python
-from pyspark.sql import SparkSession
import os
+from pyspark.sql import SparkSession
-gravitino_url = "http://localhost:8090";
-metalake_name = "test"
-
-catalog_name = "your_cos_catalog"
-schema_name = "your_cos_schema"
-fileset_name = "your_cos_fileset"
-
-# JDK8 as follows. JDK17 will be slightly different, you need to add
-# '--conf
"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
-# --conf
"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"'
-# to the submit args.
+# On JDK 17, also add:
+# --conf
"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
+# --conf
"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
os.environ["PYSPARK_SUBMIT_ARGS"] = (
- "--jars "
- "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar,"
+ "--jars
/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar,"
"/path/to/hadoop-cos-3.3.0-8.3.23.jar,"
"/path/to/cos_api-bundle-5.6.227.jar "
"--master local[1] pyspark-shell"
)
-spark = SparkSession.builder \
- .appName("cos_fileset_test") \
- .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs") \
- .config("spark.hadoop.fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") \
- .config("spark.hadoop.fs.gravitino.server.uri", gravitino_url) \
- .config("spark.hadoop.fs.gravitino.client.metalake", "test") \
- .config("spark.hadoop.cos-region", "ap-guangzhou") \
- .config("spark.hadoop.cos-access-key-id", os.environ["COS_ACCESS_KEY_ID"])
\
- .config("spark.hadoop.cos-secret-access-key",
os.environ["COS_SECRET_ACCESS_KEY"]) \
- .config("spark.driver.memory", "2g") \
- .config("spark.driver.port", "2048") \
- .getOrCreate()
+
+spark = (SparkSession.builder
+ .appName("cos_fileset")
+ .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs")
+ .config("spark.hadoop.fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem")
+ .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090";)
+ .config("spark.hadoop.fs.gravitino.client.metalake", "metalake")
+ .config("spark.hadoop.cos-region", "ap-guangzhou")
+ .config("spark.hadoop.cos-endpoint", "cos.ap-guangzhou.myqcloud.com")
+ .config("spark.hadoop.cos-access-key-id", "access_key")
+ .config("spark.hadoop.cos-secret-access-key", "secret_key")
+ .config("spark.driver.memory", "2g")
+ .config("spark.driver.port", "2048")
+ .getOrCreate())
data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)]
-columns = ["Name", "Age"]
-spark_df = spark.createDataFrame(data, schema=columns)
-gvfs_path =
f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people"
-
-spark_df.coalesce(1).write \
- .mode("overwrite") \
- .option("header", "true") \
- .csv(gvfs_path)
+spark_df = spark.createDataFrame(data, schema=["Name", "Age"])
+gvfs_path = "gvfs://fileset/cos_catalog/cos_schema/example_fileset/people"
+
+spark_df.coalesce(1).write.mode("overwrite").option("header",
"true").csv(gvfs_path)
```
-If your Spark is **without Hadoop environment**, you can use the following
code snippet to access the fileset:
+If Spark runs without a Hadoop environment, only the jar list changes:
```python
-## Only the PYSPARK_SUBMIT_ARGS line below changes; keep all the
SparkSession.builder.config(...)
-## calls (including spark.hadoop.cos-region / cos-access-key-id /
cos-secret-access-key) above as-is.
-
-os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars
/path/to/gravitino-tencent-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar
--master local[1] pyspark-shell"
+os.environ["PYSPARK_SUBMIT_ARGS"] = (
+ "--jars /path/to/gravitino-tencent-bundle-${gravitino-version}.jar,"
+ "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar "
+ "--master local[1] pyspark-shell"
+)
```
--
[`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle):
A "fat" JAR that includes `gravitino-tencent` functionality and all necessary
dependencies like `hadoop-cos` and the Tencent Cloud COS Java SDK. Use this if
your Spark environment doesn't have a pre-existing Hadoop setup.
--
[`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime):
A "fat" JAR that bundles Gravitino's virtual filesystem client and includes
the functionality of `gravitino-tencent`. It is required for accessing
Gravitino filesets.
-
-Please choose the correct jar according to your environment.
-
:::note
-In some Spark versions, a Hadoop environment is needed by the driver, adding
the bundle jars with `--jars` may not work. If this is the case, you should add
the jars to the spark CLASSPATH directly.
+Some Spark versions need a Hadoop environment in the driver and do not pick up
filesystem
+implementations passed with `--jars`. If that happens, add the jars to the
Spark classpath directly.
:::
-### Access a Fileset Using the Hadoop Fs Command
-
-The following are examples of how to use the `hadoop fs` command to access the
fileset in Hadoop 3.1.3:
+### Hadoop fs command
-1. Add the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml`
file:
+1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`:
```xml
- <property>
- <name>fs.AbstractFileSystem.gvfs.impl</name>
- <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
- </property>
-
- <property>
- <name>fs.gvfs.impl</name>
-
<value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
- </property>
-
- <property>
- <name>fs.gravitino.server.uri</name>
- <value>http://localhost:8090</value>
- </property>
-
- <property>
- <name>fs.gravitino.client.metalake</name>
- <value>test</value>
- </property>
-
- <property>
- <name>cos-region</name>
- <value>ap-guangzhou</value>
- </property>
-
- <property>
- <name>cos-access-key-id</name>
- <value>access-key</value>
- </property>
-
- <property>
- <name>cos-secret-access-key</name>
- <value>secret-key</value>
- </property>
+<property>
+ <name>fs.AbstractFileSystem.gvfs.impl</name>
+ <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
+</property>
+<property>
+ <name>fs.gvfs.impl</name>
+
<value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
+</property>
+<property>
+ <name>fs.gravitino.server.uri</name>
+ <value>http://localhost:8090</value>
+</property>
+<property>
+ <name>fs.gravitino.client.metalake</name>
+ <value>metalake</value>
+</property>
+<property>
+ <name>cos-region</name>
+ <value>ap-guangzhou</value>
+</property>
+<property>
+ <name>cos-endpoint</name>
+ <value>cos.ap-guangzhou.myqcloud.com</value>
+</property>
+<property>
+ <name>cos-access-key-id</name>
+ <value>access_key</value>
+</property>
+<property>
+ <name>cos-secret-access-key</name>
+ <value>secret_key</value>
+</property>
```
-2. Add the necessary jars to the Hadoop classpath.
-
-For COS, you need to add
`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`,
`hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar` to the Hadoop
classpath. Unlike `hadoop-aws` or `hadoop-aliyun`, these jars are *not* part of
the Apache Hadoop distribution; download them from Maven Central and place them
under `${HADOOP_HOME}/share/hadoop/tools/lib/`.
+2. Add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar` and
`hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar` — unlike
`hadoop-aws` or `hadoop-aliyun` these are not part of the Apache Hadoop
distribution, download them from Maven Central
+ to the Hadoop classpath.
-3. Run the following command to access the fileset:
+3. Access the fileset:
```shell
-./${HADOOP_HOME}/bin/hadoop fs -ls
gvfs://fileset/cos_catalog/cos_schema/cos_fileset
-./${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/cos_catalog/cos_schema/cos_fileset
+${HADOOP_HOME}/bin/hadoop fs -ls
gvfs://fileset/cos_catalog/cos_schema/example_fileset
+${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/cos_catalog/cos_schema/example_fileset
```
-### Access the Fileset with the GVFS Python Client / Pandas
+### GVFS Python client
-:::note
-The GVFS **Python** client does not yet ship a COS storage handler, so reading
-and writing COS-backed filesets via the Python
`gvfs.GravitinoVirtualFileSystem`
-or pandas `read_csv("gvfs://...")` is **not supported in this release**. Only
-the GVFS **Java** client (and Spark / `hadoop fs` on top of it) can do fileset
-I/O against COS today.
-
-The Python `GravitinoClient` itself is not affected: you can still create,
-inspect, update and delete COS catalogs, schemas and filesets through the
-metadata API exactly as shown in the [Step 1 example
above](#step-1-create-a-fileset-catalog-with-cos) and the following Step 2 /
Step 3 sections.
-What is missing is purely the *data plane* (`gvfs.ls`, `gvfs.open`,
-pandas `read_csv("gvfs://...")`, etc.) for COS, which will be added in a
-follow-up release together with a `COSStorageHandler` and the corresponding
-fsspec adapter.
-:::
+```bash
+pip install apache-gravitino==${GRAVITINO_VERSION}
+```
-For other use cases of the Java GVFS client, refer to the [Gravitino Virtual
File System](./how-to-use-gvfs.md) document.
+On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Tencent Cloud COS
+properties in `options`, spelled with underscores.
Review Comment:
These examples imply the Python GVFS client can access COS-backed filesets,
but the Python implementation currently cannot resolve `cosn://` storage
locations: `clients/client-python/gravitino/filesystem/gvfs_storage_handler.py`
does not define a COS/COSN `StorageType` or prefix mapping, and
`get_storage_handler_by_path` will raise `Storage type doesn't support now` for
`cosn://...` paths. The COS page should either restore the prior “Java-only”
limitation for Python/pandas or add COS support to the Python client before
documenting it as runnable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
