tengqm commented on code in PR #6071:
URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903382830
##########
docs/iceberg-rest-service.md:
##########
@@ -160,40 +145,28 @@ Supports using static GCS credential file or generating
GCS token to access GCS
| Configuration item | Description
|
Default value | Required | Since Version |
|---------------------------------------------------|----------------------------------------------------------------------------------------------------|---------------|----------|------------------|
| `gravitino.iceberg-rest.io-impl` | The io implementation
for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. |
(none) | No | 0.6.0-incubating |
-| `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use
`gravitino.iceberg-rest.credential-providers` instead. |
(none) | No |
0.7.0-incubating |
-| `gravitino.iceberg-rest.credential-providers` | Supports `gcs-token`,
generates a temporary token according to the query data path. |
(none) | No | 0.7.0-incubating |
-| `gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use
`gravitino.iceberg-rest.gcs-service-account-file` instead. |
(none) | No | 0.7.0-incubating |
-| `gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS
credential file, only used when `credential-provider-type` is `gcs-token`.
| (none) | No | 0.8.0-incubating |
For other Iceberg GCS properties not managed by Gravitino like
`gcs.project-id`, you could config it directly by
`gravitino.iceberg-rest.gcs.project-id`.
-If you set `credential-providers` explicitly, please downloading [Gravitino
GCP bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle), and
place it to the classpath of Iceberg REST server.
+Please refer to [GCS
credentials](./security/credential-vending.md#gcs-credentials) for credential
related configurations.
-Please make sure the credential file is accessible by Gravitino, like using
`export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` before
Gravitino Iceberg REST server is started.
+:::note
+For Gravitino Iceberg REST server, please make sure the credential file is
accessible by Gravitino, like using `export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even
`gcs-service-account-file` is setting.
Review Comment:
This sentence is not easy to parse.
Please consider revise it.
##########
docs/security/credential-vending.md:
##########
@@ -0,0 +1,178 @@
+---
+title: "Gravitino credential vending"
+slug: /security/credential-vending
+keyword: security credential vending
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Background
+
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
+
+### Capabilities
+
+- Supports Gravitino Iceberg REST server.
+- Supports Gravitino server, only support Hadoop catalog.
+- Supports pluggable credentials with build-in credentials:
+ - S3: `S3TokenCredential`, `S3SecretKeyCredential`
+ - GCS: `GCSTokenCredential`
+ - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
+ - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
+- Doesn't support Spark/Trino/Flink connector.
Review Comment:
```suggestion
- No support to Spark/Trino/Flink connector yet.
```
##########
docs/security/credential-vending.md:
##########
@@ -0,0 +1,178 @@
+---
+title: "Gravitino credential vending"
+slug: /security/credential-vending
+keyword: security credential vending
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Background
+
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
+
+### Capabilities
+
+- Supports Gravitino Iceberg REST server.
+- Supports Gravitino server, only support Hadoop catalog.
+- Supports pluggable credentials with build-in credentials:
+ - S3: `S3TokenCredential`, `S3SecretKeyCredential`
+ - GCS: `GCSTokenCredential`
+ - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
+ - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
+- Doesn't support Spark/Trino/Flink connector.
+
+## General configurations
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since
Version |
+|-------------------------------------|--------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-provider-type` |
`gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use
`credential-providers` instead. | (none)
| Yes | 0.7.0-incubating |
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | The credential
provider types, separated by comma. |
(none) | Yes | 0.8.0-incubating |
+| `credential-cache-expire-ratio` |
`gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the
credential's expiration time when Gravitino remove credential from the cache. |
0.15 | No | 0.8.0-incubating |
+| `credential-cache-max-size` |
`gravitino.iceberg-rest.cache-max-size` | Max size for the
credential cache. |
10000 | No | 0.8.0-incubating |
+
+## Build-in credentials configurations
+
+### S3 credentials
+
+#### S3 secret key credential
+
+A credential with static S3 access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description |
Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|--------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3
secret key credential provider. | (none) | Yes | 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+
+#### S3 token credential
+
+A credential with a subscope privileged token, to use S3 token credential, you
should create a role and grant proper privilege to the role to access the S3
data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token
credential provider.
| (none) | Yes
| 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the role to access the S3 data.
| (none) | Yes | 0.7.0-incubating |
+| `s3-external-id` |
`gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token.
| (none) | No
| 0.7.0-incubating |
+| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role. | 3600 | No
| 0.7.0-incubating |
+| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative endpoint of
the S3 token service, This could be used with s3-compatible object storage
service like MINIO that has a different STS endpoint. | (none) | No
| 0.8.0-incubating |
+
+### OSS credentials
+
+#### OSS secret key credential
+
+A credential with static OSS access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-secret-key` for OSS
secret credential. | (none) | Yes
| 0.8.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+
+#### OSS token credential
+
+A credential with a subscope privileged token, to use OSS token credential,
you should create an role and grant proper privilege to the role to access the
OSS data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-token` for s3 token
credential. | (none) | Yes
| 0.8.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-role-arn` | `gravitino.iceberg-rest.oss-role-arn`
| The ARN of the role to access the OSS data.
| (none) | Yes | 0.8.0-incubating |
+| `oss-external-id` |
`gravitino.iceberg-rest.oss-external-id` | The OSS external id to
generate token. | (none) | No
| 0.8.0-incubating |
+| `oss-token-expire-in-secs` |
`gravitino.iceberg-rest.oss-token-expire-in-secs` | The OSS security token
expire time in secs. | 3600 | No
| 0.8.0-incubating |
+
+### ADLS credentials
+
+#### Azure account key credential
+
+A credential with static Azure storage account name and key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|-----------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `azure-account-key` for
Azure account key credential. | (none) | Yes | 0.8.0-incubating
|
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+
+#### ADLS token credential
+
+A credential with a subscope privileged token, to use ADLS token credential,
you should create an Microsoft Entra ID service principal and grant proper
privilege to it to access the Azure data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `adls-token` for ADLS
token credential. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-tenant-id` |
`gravitino.iceberg-rest.azure-tenant-id` | Azure Active Directory
(AAD) tenant ID. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-id` |
`gravitino.iceberg-rest.azure-client-id` | Azure Active Directory
(AAD) client ID used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-secret` |
`gravitino.iceberg-rest.azure-client-secret` | Azure Active Directory
(AAD) client secret used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `adls-token-expire-in-secs` |
`gravitino.iceberg-rest.adls-token-expire-in-secs` | The ADLS SAS token expire
time in secs. | 3600 | No |
0.8.0-incubating |
+
+### GCS credentials
+
+#### GCS token credential
+
+A credential with a subscope privileged token, to use GCS token credential,
you should create an GCS service account and grant proper privilege to it to
access the OSS data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|------------------------------------------------------------|-------------------------------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `gcs-token` for GCS token
credential. | (none) | Yes
| 0.8.0-incubating |
+| `gcs-credential-file-path` |
`gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use
`gcs-service-account-file` instead. | GCS Application default credential. | No
| 0.7.0-incubating |
+| `gcs-service-account-file` |
`gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS
credential file. | GCS Application default credential. |
No | 0.8.0-incubating |
+
+:::note
+For Gravitino Iceberg REST server, please make sure the credential file is
accessible by Gravitino, like using `export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even
`gcs-service-account-file` is setting.
+:::
Review Comment:
Please tweak this sentence to make sense.
##########
docs/security/credential-vending.md:
##########
@@ -0,0 +1,178 @@
+---
+title: "Gravitino credential vending"
+slug: /security/credential-vending
+keyword: security credential vending
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Background
+
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
+
+### Capabilities
+
+- Supports Gravitino Iceberg REST server.
+- Supports Gravitino server, only support Hadoop catalog.
+- Supports pluggable credentials with build-in credentials:
+ - S3: `S3TokenCredential`, `S3SecretKeyCredential`
+ - GCS: `GCSTokenCredential`
+ - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
+ - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
+- Doesn't support Spark/Trino/Flink connector.
+
+## General configurations
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since
Version |
+|-------------------------------------|--------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-provider-type` |
`gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use
`credential-providers` instead. | (none)
| Yes | 0.7.0-incubating |
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | The credential
provider types, separated by comma. |
(none) | Yes | 0.8.0-incubating |
+| `credential-cache-expire-ratio` |
`gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the
credential's expiration time when Gravitino remove credential from the cache. |
0.15 | No | 0.8.0-incubating |
+| `credential-cache-max-size` |
`gravitino.iceberg-rest.cache-max-size` | Max size for the
credential cache. |
10000 | No | 0.8.0-incubating |
+
+## Build-in credentials configurations
+
+### S3 credentials
+
+#### S3 secret key credential
+
+A credential with static S3 access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description |
Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|--------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3
secret key credential provider. | (none) | Yes | 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+
+#### S3 token credential
+
+A credential with a subscope privileged token, to use S3 token credential, you
should create a role and grant proper privilege to the role to access the S3
data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token
credential provider.
| (none) | Yes
| 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the role to access the S3 data.
| (none) | Yes | 0.7.0-incubating |
+| `s3-external-id` |
`gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token.
| (none) | No
| 0.7.0-incubating |
+| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role. | 3600 | No
| 0.7.0-incubating |
+| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative endpoint of
the S3 token service, This could be used with s3-compatible object storage
service like MINIO that has a different STS endpoint. | (none) | No
| 0.8.0-incubating |
+
+### OSS credentials
+
+#### OSS secret key credential
+
+A credential with static OSS access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-secret-key` for OSS
secret credential. | (none) | Yes
| 0.8.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+
+#### OSS token credential
+
+A credential with a subscope privileged token, to use OSS token credential,
you should create an role and grant proper privilege to the role to access the
OSS data.
Review Comment:
```suggestion
An OSS token is a token credential with scoped privileges.
To use an OSS token credential, you should create a role and grant it proper
privileges.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
