yuqi1129 commented on code in PR #6071:
URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903615861
##########
docs/security/credential-vending.md:
##########
@@ -0,0 +1,178 @@
+---
+title: "Gravitino credential vending"
+slug: /security/credential-vending
+keyword: security credential vending
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Background
+
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
+
+### Capabilities
+
+- Supports Gravitino Iceberg REST server.
+- Supports Gravitino server, only support Hadoop catalog.
+- Supports pluggable credentials with build-in credentials:
+ - S3: `S3TokenCredential`, `S3SecretKeyCredential`
+ - GCS: `GCSTokenCredential`
+ - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
+ - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
+- No support to Spark/Trino/Flink connector yet.
+
+## General configurations
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since
Version |
+|-------------------------------------|--------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-provider-type` |
`gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use
`credential-providers` instead. | (none)
| Yes | 0.7.0-incubating |
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | The credential
provider types, separated by comma. |
(none) | Yes | 0.8.0-incubating |
+| `credential-cache-expire-ratio` |
`gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the
credential's expiration time when Gravitino remove credential from the cache. |
0.15 | No | 0.8.0-incubating |
+| `credential-cache-max-size` |
`gravitino.iceberg-rest.cache-max-size` | Max size for the
credential cache. |
10000 | No | 0.8.0-incubating |
+
+## Build-in credentials configurations
+
+### S3 credentials
+
+#### S3 secret key credential
+
+A credential with static S3 access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description |
Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|--------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3
secret key credential provider. | (none) | Yes | 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+
+#### S3 token credential
+
+An S3 token is a token credential with scoped privileges. To use an S3 token
credential, you should create a role and grant it proper privileges.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token
credential provider.
| (none) | Yes
| 0.8.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the role to access the S3 data.
| (none) | Yes | 0.7.0-incubating |
+| `s3-external-id` |
`gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token.
| (none) | No
| 0.7.0-incubating |
+| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role. | 3600 | No
| 0.7.0-incubating |
+| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative endpoint of
the S3 token service, This could be used with s3-compatible object storage
service like MINIO that has a different STS endpoint. | (none) | No
| 0.8.0-incubating |
+
+### OSS credentials
+
+#### OSS secret key credential
+
+A credential with static OSS access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-secret-key` for OSS
secret credential. | (none) | Yes
| 0.8.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+
+#### OSS token credential
+
+An OSS token is a token credential with scoped privileges. To use an OSS token
credential, you should create a role and grant it proper privileges.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-token` for s3 token
credential. | (none) | Yes
| 0.8.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-role-arn` | `gravitino.iceberg-rest.oss-role-arn`
| The ARN of the role to access the OSS data.
| (none) | Yes | 0.8.0-incubating |
+| `oss-external-id` |
`gravitino.iceberg-rest.oss-external-id` | The OSS external id to
generate token. | (none) | No
| 0.8.0-incubating |
+| `oss-token-expire-in-secs` |
`gravitino.iceberg-rest.oss-token-expire-in-secs` | The OSS security token
expire time in secs. | 3600 | No
| 0.8.0-incubating |
+
+### ADLS credentials
+
+#### Azure account key credential
+
+A credential with static Azure storage account name and key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|-----------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `azure-account-key` for
Azure account key credential. | (none) | Yes | 0.8.0-incubating
|
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+
+#### ADLS token credential
+
+An ADLS token is a token credential with scoped privileges. To use an ADLS
token credential, you should create a Microsoft Entra ID service principal and
grant it proper privileges.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `adls-token` for ADLS
token credential. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-tenant-id` |
`gravitino.iceberg-rest.azure-tenant-id` | Azure Active Directory
(AAD) tenant ID. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-id` |
`gravitino.iceberg-rest.azure-client-id` | Azure Active Directory
(AAD) client ID used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-secret` |
`gravitino.iceberg-rest.azure-client-secret` | Azure Active Directory
(AAD) client secret used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `adls-token-expire-in-secs` |
`gravitino.iceberg-rest.adls-token-expire-in-secs` | The ADLS SAS token expire
time in secs. | 3600 | No |
0.8.0-incubating |
+
+### GCS credentials
+
+#### GCS token credential
+
+An GCS token is a token credential with scoped privileges. To use an GCS token
credential, you should create an GCS service account and grant it proper
privileges.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|------------------------------------------------------------|-------------------------------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `gcs-token` for GCS token
credential. | (none) | Yes
| 0.8.0-incubating |
+| `gcs-credential-file-path` |
`gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use
`gcs-service-account-file` instead. | GCS Application default credential. | No
| 0.7.0-incubating |
+| `gcs-service-account-file` |
`gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS
credential file. | GCS Application default credential. |
No | 0.8.0-incubating |
+
+:::note
+For Gravitino Iceberg REST server, please ensure that the credential file can
be accessed by the server. For example, if the server is running on a GCE
machine, or you can set the environment variable as `export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`, even
when the `gcs-service-account-file` has already been configured.
+:::
+
+## Custom credentials
+
+Gravitino supports custom credentials, you can implement the
`org.apache.gravitino.credential.CredentialProvider` interface to support
custom credentials, and place the corresponding jar to the classpath of Iceberg
catalog server or Hadoop catalog.
+
+## Deployment
+
+Besides setting credentials related configuration, please download Gravitino
cloud bundle jar and place it in the classpath of Iceberg catalog server or
Hadoop catalog.
+
+Gravitino cloud bundle jar:
+
+- [Gravitino AWS bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/aws-bundle)
+- [Gravitino Aliyun bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/aliyun-bundle)
+- [Gravitino GCP bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle)
+- [Gravitino Azure bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/azure-bundle)
+
+The classpath of the server:
+
+- Iceberg REST server: the classpath diffs in different deploy mode, please
refer to [Server manage](../iceberg-rest-service.md#server-management) part.
+- Hadoop catalog: `catalogs/hadoop/libs/`
+
+## Scenarios
+
+### Credential vending for Iceberg REST server
+
+Suppose the Iceberg table data is stored in S3, follow the steps below:
+
+1. Download the Gravitino AWS bundle jar, and place it to the classpath of
Iceberg REST server.
Review Comment:
Add a link here(AWS bundle jar).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
