orenccl commented on code in PR #6071:
URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901403063
##########
docs/security/credential-vending.md:
##########
@@ -0,0 +1,175 @@
+---
+title: "Gravitino credential vending"
+slug: /security/credential-vending
+keyword: security credential vending
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Background
+
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
+
+### Capabilities
+
+- Supports Gravitino Iceberg REST server.
+- Supports Gravitino server, only support Hadoop catalog.
+- Doesn't support Spark/Trino/Flink connector.
+- With Build-in credentials:
+ - S3: `S3TokenCredential`, `S3SecretKeyCredential`
+ - GCS: `GCSTokenCredential`
+ - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
+ - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
+
+## General configurations
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since
Version |
+|-------------------------------------|--------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-provider-type` |
`gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use
`credential-providers` instead. | (none)
| Yes | 0.7.0-incubating |
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | The credential
provider types, separated by comma. |
(none) | Yes | 0.8.0-incubating |
+| `credential-cache-expire-ratio` |
`gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the
credential's expiration time when Gravitino remove credential from the cache. |
0.15 | No | 0.8.0-incubating |
+| `credential-cache-max-size` |
`gravitino.iceberg-rest.cache-max-size` | Max size for the
credential cache. |
10000 | No | 0.8.0-incubating |
+
+## Build-in credentials configurations
+
+### S3 credentials
+
+#### S3 secret key credential
+
+A credential with static S3 access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description |
Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|--------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3
secret key credential provider. | (none) | Yes | 0.7.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+
+#### S3 token credential
+
+A credential with a subscope privileged token, to use S3 token credential, you
should create a role and grant proper privilege to the role to access the S3
data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token
credential provider.
| (none) | Yes
| 0.7.0-incubating |
+| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
+| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the role to access the S3 data.
| (none) | Yes | 0.7.0-incubating |
+| `s3-external-id` |
`gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token.
| (none) | No
| 0.7.0-incubating |
+| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role. | 3600 | No
| 0.7.0-incubating |
+| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative endpoint of
the S3 token service, This could be used with s3-compatible object storage
service like MINIO that has a different STS endpoint. | (none) | No
| 0.8.0-incubating |
+
+### OSS credentials
+
+#### OSS secret key credential
+
+A credential with static OSS access key id and secret access key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-secret-key` for OSS
secret credential. | (none) | Yes
| 0.7.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+
+#### OSS token credential
+
+A credential with a subscope privileged token, to use OSS token credential,
you should create an role and grant proper privilege to the role to access the
OSS data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-token` for s3 token
credential. | (none) | Yes
| 0.7.0-incubating |
+| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+| `oss-role-arn` | `gravitino.iceberg-rest.oss-role-arn`
| The ARN of the role to access the OSS data.
| (none) | Yes | 0.8.0-incubating |
+| `oss-external-id` |
`gravitino.iceberg-rest.oss-external-id` | The OSS external id to
generate token. | (none) | No
| 0.8.0-incubating |
+| `oss-token-expire-in-secs` |
`gravitino.iceberg-rest.oss-token-expire-in-secs` | The OSS security token
expire time in secs. | 3600 | No
| 0.8.0-incubating |
+
+### ADLS credentials
+
+#### Azure account key credential
+
+A credential with static Azure storage account name and key.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|-----------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `azure-account-key` for
Azure account key credential. | (none) | Yes | 0.8.0-incubating
|
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+
+#### ADLS token credential
+
+A credential with a subscope privileged token, to use OSS token credential,
you should create an Microsoft Entra ID service principal and grant proper
privilege to it to access the Azure data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
+|-------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------|---------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `adls-token` for ADLS
token credential. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+| `azure-tenant-id` |
`gravitino.iceberg-rest.azure-tenant-id` | Azure Active Directory
(AAD) tenant ID. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-id` |
`gravitino.iceberg-rest.azure-client-id` | Azure Active Directory
(AAD) client ID used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `azure-client-secret` |
`gravitino.iceberg-rest.azure-client-secret` | Azure Active Directory
(AAD) client secret used for authentication. | (none) | Yes |
0.8.0-incubating |
+| `adls-token-expire-in-secs` |
`gravitino.iceberg-rest.adls-token-expire-in-secs` | The ADLS SAS token expire
time in secs. | 3600 | No |
0.8.0-incubating |
+
+### GCS credentials
+
+#### GCS token credential
+
+A credential with a subscope privileged token, to use GCS token credential,
you should create an GCS service account and grant proper privilege to it to
access the OSS data.
+
+| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description | Default value
| Required | Since Version |
+|-------------------------------------|---------------------------------------------------|---------------------------------------|-------------------------------------|----------|------------------|
+| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `gcs-token` for GCS token
credential. | (none) | Yes | 0.7.0-incubating
|
+| `gcs-credential-file-path` |
`gravitino.iceberg-rest.gcs-credential-file-path` | The location of GCS
credential file. | GCS Application default credential. | No |
0.7.0-incubating |
+
+Please make sure the credential file is accessible by Gravitino, like using
`export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` before
Gravitino Iceberg REST server is started.
+
+## Custom credentials
+
+Gravitino supports custom credentials, you can implement the
`org.apache.gravitino.credential.CredentialProvider` interface to support
custom credentials, and place the corresponding jar to the classpath of Iceberg
catalog server or hadoop catalog.
+
+## Deployment
+
+Besides set credential related configuration, please downloading Gravitino
cloud jar and place it to the classpath of Iceberg catalog server or hadoop
catalog.
+
+Gravitino cloud jar:
+
+- [Gravitino AWS bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/aws-bundle)
+- [Gravitino Aliyun bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/aliyun-bundle)
+- [Gravitino GCP bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle)
+- [Gravitino Azure bundle
jar](https://mvnrepository.com/artifact/org.apache.gravitino/azure-bundle)
+
+The classpath of the server:
+
+- Iceberg REST server: the classpath diffs in different deploy mode, please
refer to [Server manage](../iceberg-rest-service.md#server-management) part.
+- hadoop catalog: `catalogs/hadoop/libs/`
+
+## Scenarios
+
+### Credential vending for Iceberg REST server
Review Comment:
“My bad, I mistook ‘aws’ for ‘Azure’.”
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
