yuqi1129 commented on code in PR #5324:
URL: https://github.com/apache/gravitino/pull/5324#discussion_r1820713863
##########
docs/hadoop-catalog.md:
##########
@@ -25,19 +25,30 @@ Hadoop 3. If there's any compatibility issue, please create
an [issue](https://g
Besides the [common catalog
properties](./gravitino-server-config.md#gravitino-catalog-properties-configuration),
the Hadoop catalog has the following properties:
-| Property Name | Description
| Default Value | Required
| Since Version |
-|----------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|-------------------------------------------------------------|------------------|
-| `location` | The storage location
managed by Hadoop catalog.
| (none) | No
| 0.5.0 |
-| `filesystem-providers` | The names (split by
comma) of filesystem providers for the Hadoop catalog. Gravitino already
support built-in `builtin-local`(`local file`) and `builtin-hdfs`(`hdfs`). If
users want to support more file system and add it to Gravitino, they custom
more file system by implementing `FileSystemProvider`. | (none) | No
| 0.7.0-incubating |
-| `default-filesystem-provider` | The name default
filesystem providers of this Hadoop catalog if users do not specify the scheme
in the URI. Default value is `builtin-local`
| `builtin-local` | No
| 0.7.0-incubating |
-| `authentication.impersonation-enable` | Whether to enable
impersonation for the Hadoop catalog.
| `false` | No
| 0.5.1 |
-| `authentication.type` | The type of
authentication for Hadoop catalog, currently we only support `kerberos`,
`simple`.
| `simple` |
No | 0.5.1 |
-| `authentication.kerberos.principal` | The principal of the
Kerberos authentication
| (none) | required if the
value of `authentication.type` is Kerberos. | 0.5.1 |
-| `authentication.kerberos.keytab-uri` | The URI of The keytab
for the Kerberos authentication.
| (none) | required if the
value of `authentication.type` is Kerberos. | 0.5.1 |
-| `authentication.kerberos.check-interval-sec` | The check interval of
Kerberos credential for Hadoop catalog.
| 60 | No
| 0.5.1 |
-| `authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of
retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`.
| 60 | No
| 0.5.1 |
-
-For more about `filesystem-providers`, please refer to
`HadoopFileSystemProvider` or `LocalFileSystemProvider` in the source code.
Furthermore, you also need to place the jar of the file system provider into
the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory if it's not in the
classpath.
+| Property Name | Description
| Default Value | Required
| Since Version |
+|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|-------------------------------------------------------------|------------------|
+| `location` | The storage location
managed by Hadoop catalog.
| (none) | No
| 0.5.0 |
+| `filesystem-providers` | The names (split by
comma) of filesystem providers for the Hadoop catalog. Gravitino already
support built-in `builtin-local`(`local file`) and `builtin-hdfs`(`hdfs`). If
users want to support more file system and add it to Gravitino, they custom
more file system by implementing `FileSystemProvider`. | (none) | Yes
if use s3, gcs or oss. | 0.7.0-incubating |
+| `default-filesystem-provider` | The name default
filesystem providers of this Hadoop catalog if users do not specify the scheme
in the URI. Default value is `builtin-local`
| `builtin-local` | No
| 0.7.0-incubating |
+| `authentication.impersonation-enable` | Whether to enable
impersonation for the Hadoop catalog.
| `false` | No
| 0.5.1 |
+| `authentication.type` | The type of
authentication for Hadoop catalog, currently we only support `kerberos`,
`simple`.
| `simple` | No
| 0.5.1 |
+| `authentication.kerberos.principal` | The principal of the
Kerberos authentication
| (none) | required if the
value of `authentication.type` is Kerberos. | 0.5.1 |
+| `authentication.kerberos.keytab-uri` | The URI of The keytab
for the Kerberos authentication.
| (none) | required if the
value of `authentication.type` is Kerberos. | 0.5.1 |
+| `authentication.kerberos.check-interval-sec` | The check interval of
Kerberos credential for Hadoop catalog.
| 60 | No
| 0.5.1 |
+| `authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of
retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`.
| 60 | No
| 0.5.1 |
+| `s3-endpoint` | The endpoint of the AWS
s3.
| (none) | Yes if use s3
| 0.7.0-incubating |
+| `s3-access-key-id` | The access key of the
AWS s3.
| (none) | Yes if use s3
| 0.7.0-incubating |
+| `s3-secret-access-key` | The secret key of the
AWS s3.
| (none) | Yes if use s3
| 0.7.0-incubating |
+| `gcs-service-account-file` | The path of GCS service
account JSON file.
| (none) | Yes if use gcs
| 0.7.0-incubating |
+| `oss-endpoint` | The endpoint of the
Aliyun oss.
| (none) | Yes if use oss
| 0.7.0-incubating |
+| `oss-access-key-id` | The access key of the
Aliyun oss.
| (none) | Yes if use oss
| 0.7.0-incubating |
+| `oss-secret-access-key` | The secret key of the
Aliyun oss.
| (none) | Yes if use oss
| 0.7.0-incubating |
+
+:::warning
+Since Gravitino 0.7.0-incubating, the Hadoop catalog supports S3, GCS, and
OSS. To enable the support for these object storages, you need to put the
corresponding bundle jar like `gravitino-aws-bundle-{version}.jar`,
`gravitino-gcs-bundle-{version}.jar`, `gravitino-oss-bundle-{version}.jar`
+into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory and set the
corresponding configurations.
+:::
Review Comment:
I was hesitate to add it separately as some properties like
`filesystem-providers` are shared by several cloud storage, what about adding
the following content here?
```markdown
### Properties required to create a local file system Hadoop Catalog
No additional properties are required to create a local file system Hadoop
catalog.
### Properties required to create an HDFS Hadoop Catalog
Similarly, no additional properties are required to create an HDFS Hadoop
catalog if the target HDFS cluster is not Kerberos-enabled. If the target HDFS
cluster is Kerberos-enabled, the following properties are required:
- `authentication.type`: The type of authentication for the Hadoop catalog.
The value should be `kerberos`.
- `authentication.kerberos.principal`: The principal of the Kerberos
authentication.
- `authentication.kerberos.keytab-uri`: The URI of the keytab for the
Kerberos authentication.
### Properties required to create an S3 Hadoop Catalog
To create an S3 Hadoop catalog, you need to set the following properties:
- `filesystem-providers`: The value should be `s3` or a comma-separated list
of that contains `s3` like `builtin-local,s3`.
- `s3-endpoint`: The endpoint of the AWS S3.
- `s3-access-key-id`: The access key of the AWS S3.
- `s3-secret-access-key`: The secret key of the AWS S3.
At the same time, you need to put the corresponding bundle jar
`gravitino-aws-bundle-{version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory.
### Properties required to create a GCS Hadoop Catalog
To create a GCS Hadoop catalog, you need to set the following properties:
- `filesystem-providers`: The value should be `gcs` or a comma-separated
list of that contains `gcs` like `builtin-local,gcs`.
- `gcs-service-account-file`: The path of the GCS service account JSON file.
At the same time, you need to put the corresponding bundle jar
`gravitino-gcs-bundle-{version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory.
### Properties required to create an OSS Hadoop Catalog
To create an OSS Hadoop catalog, you need to set the following properties:
- `filesystem-providers`: The value should be `oss` or a comma-separated
list of that contains `oss` like `builtin-local,oss`.
- `oss-endpoint`: The endpoint of the Aliyun OSS.
- `oss-access-key-id`: The access key of the Aliyun OSS.
- `oss-secret-access-key`: The secret key of the Aliyun OSS.
At the same time, you need to put the corresponding bundle jar
`gravitino-oss-bundle-{version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
