Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao merged PR #6071: URL: https://github.com/apache/gravitino/pull/6071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904217626 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. Review Comment: updated ## docs/hadoop-catalog.md: ## @@ -166,6 +178,13 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +Credential providers can be specified in several places, as listed below. Gravatino checks the provider settings in the following order of precedence: Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904217082 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges, by leveraging STS [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The stati
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904217411 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges, by leveraging STS [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The stati
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904105722 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges, by leveraging STS [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The st
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904110418 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges, by leveraging STS [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The st
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904106249 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges, by leveraging STS [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The st
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904098211 ## docs/hadoop-catalog.md: ## @@ -166,6 +178,13 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +Credential providers can be specified in several places, as listed below. Gravatino checks the provider settings in the following order of precedence: Review Comment: checks the `credential-providers` setting... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904096809 ## docs/hadoop-catalog.md: ## @@ -166,6 +178,13 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +Credential providers can be specified in several places, as listed below. Gravatino checks the provider settings in the following order of precedence: Review Comment: Typo: Gravitino -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
jerryshao commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904102274 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. Review Comment: “No support for...” -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904027737 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges. To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1904027960 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges. To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
yuqi1129 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903615861 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges. To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
yuqi1129 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903614201 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- No support to Spark/Trino/Flink connector yet. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +An S3 token is a token credential with scoped privileges. To use an S3 token credential, you should create a role and grant it proper privileges. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903609558 ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- Doesn't support Spark/Trino/Flink connector. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.8.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903609448 ## docs/iceberg-rest-service.md: ## @@ -160,40 +145,28 @@ Supports using static GCS credential file or generating GCS token to access GCS | Configuration item| Description | Default value | Required | Since Version| |---||---|--|--| | `gravitino.iceberg-rest.io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. | (none)| No | 0.6.0-incubating | -| `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `gravitino.iceberg-rest.credential-providers` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.credential-providers` | Supports `gcs-token`, generates a temporary token according to the query data path.| (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use `gravitino.iceberg-rest.gcs-service-account-file` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS credential file, only used when `credential-provider-type` is `gcs-token`. | (none)| No | 0.8.0-incubating | For other Iceberg GCS properties not managed by Gravitino like `gcs.project-id`, you could config it directly by `gravitino.iceberg-rest.gcs.project-id`. -If you set `credential-providers` explicitly, please downloading [Gravitino GCP bundle jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle), and place it to the classpath of Iceberg REST server. +Please refer to [GCS credentials](./security/credential-vending.md#gcs-credentials) for credential related configurations. -Please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` before Gravitino Iceberg REST server is started. +:::note +For Gravitino Iceberg REST server, please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even `gcs-service-account-file` is setting. Review Comment: updated ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- Doesn't support Spark/Trino/Flink connector. Review Comment: updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
yuqi1129 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903607564 ## docs/iceberg-rest-service.md: ## @@ -160,40 +145,28 @@ Supports using static GCS credential file or generating GCS token to access GCS | Configuration item| Description | Default value | Required | Since Version| |---||---|--|--| | `gravitino.iceberg-rest.io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. | (none)| No | 0.6.0-incubating | -| `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `gravitino.iceberg-rest.credential-providers` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.credential-providers` | Supports `gcs-token`, generates a temporary token according to the query data path.| (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use `gravitino.iceberg-rest.gcs-service-account-file` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS credential file, only used when `credential-provider-type` is `gcs-token`. | (none)| No | 0.8.0-incubating | For other Iceberg GCS properties not managed by Gravitino like `gcs.project-id`, you could config it directly by `gravitino.iceberg-rest.gcs.project-id`. -If you set `credential-providers` explicitly, please downloading [Gravitino GCP bundle jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle), and place it to the classpath of Iceberg REST server. +Please refer to [GCS credentials](./security/credential-vending.md#gcs-credentials) for credential related configurations. -Please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` before Gravitino Iceberg REST server is started. +:::note +For Gravitino Iceberg REST server, please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even `gcs-service-account-file` is setting. Review Comment: For Gravitino Iceberg REST server, please make sure the credential file is accessible by Gravitino server, you can use environment variants `GOOGLE_APPLICATION_CREDENTIALS` like `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` to set the value even through configuration `gcs-service-account-file` has already set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
tengqm commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1903382830 ## docs/iceberg-rest-service.md: ## @@ -160,40 +145,28 @@ Supports using static GCS credential file or generating GCS token to access GCS | Configuration item| Description | Default value | Required | Since Version| |---||---|--|--| | `gravitino.iceberg-rest.io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. | (none)| No | 0.6.0-incubating | -| `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `gravitino.iceberg-rest.credential-providers` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.credential-providers` | Supports `gcs-token`, generates a temporary token according to the query data path.| (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated, please use `gravitino.iceberg-rest.gcs-service-account-file` instead. | (none)| No | 0.7.0-incubating | -| `gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS credential file, only used when `credential-provider-type` is `gcs-token`. | (none)| No | 0.8.0-incubating | For other Iceberg GCS properties not managed by Gravitino like `gcs.project-id`, you could config it directly by `gravitino.iceberg-rest.gcs.project-id`. -If you set `credential-providers` explicitly, please downloading [Gravitino GCP bundle jar](https://mvnrepository.com/artifact/org.apache.gravitino/gcp-bundle), and place it to the classpath of Iceberg REST server. +Please refer to [GCS credentials](./security/credential-vending.md#gcs-credentials) for credential related configurations. -Please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` before Gravitino Iceberg REST server is started. +:::note +For Gravitino Iceberg REST server, please make sure the credential file is accessible by Gravitino, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even `gcs-service-account-file` is setting. Review Comment: This sentence is not easy to parse. Please consider revise it. ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- Doesn't support Spark/Trino/Flink connector. Review Comment: ```suggestion - No support to Spark/Trino/Flink connector yet. ``` ## docs/security/credential-vending.md: ## @@ -0,0 +1,178 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Supports pluggable credentials with build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` +- Doesn't support Spark/Trino/Flink connector. + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901477441 ## docs/hadoop-catalog.md: ## @@ -166,6 +178,9 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +The credential providers in fileset properties will overwrite the credential providers in schema properties and catalog properties, if not set, the credential providers in schema properties will be used, if the credential providers in schema properties are not set, will use the credential providers in catalog properties. Review Comment: thanks, updated with ``` Credential providers can be specified in several places, as listed below. Gravatino checks the provider settings in the following order of precedence: 1. Fileset properties 2. Schema properties 3. Catalog properties ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
tengqm commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901460366 ## docs/hadoop-catalog.md: ## @@ -166,6 +178,9 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +The credential providers in fileset properties will overwrite the credential providers in schema properties and catalog properties, if not set, the credential providers in schema properties will be used, if the credential providers in schema properties are not set, will use the credential providers in catalog properties. Review Comment: This can be further shortened to something like: ``` Credential providers can be specified in several places, listed below. Gravatino checks the provider settings in the following order of precedence: 1. Fileset properties 2. Schema properties 3. Catalog properties ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901455459 ## docs/hadoop-catalog.md: ## @@ -166,6 +178,9 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema- | `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | | `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating | +| `credential-providers`| The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating | + +The credential providers in fileset properties will overwrite the credential providers in schema properties and catalog properties, if not set, the credential providers in schema properties will be used, if the credential providers in schema properties are not set, will use the credential providers in catalog properties. Review Comment: I think the original explanation might be a bit difficult to understand at first glance. How about rephrasing it like this? ```markdown Credential providers are applied in the following order of precedence: 1. **Fileset properties** override all other settings. 2. If not set in **fileset properties**, **schema properties** are used. 3. If not set in **schema properties**, **catalog properties** are used. ``` @tengqm What do you think about this version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1900957074 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | Review Comment: `credential-providers` since version should be 0.8.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901403063 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901403063 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
tengqm commented on PR #6071: URL: https://github.com/apache/gravitino/pull/6071#issuecomment-2568578677 > > By "vending", I think we mean "vendoring" ... > > `credential vending` is a specific word, please refer to https://docs.aws.amazon.com/lake-formation/latest/dg/aws-lake-formation-api-credential-vending.html or https://docs.databricks.com/en/external-access/credential-vending.html Okay. Good to learn this "new" use of "vending". Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901403063 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901403063 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901402186 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901402060 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on PR #6071: URL: https://github.com/apache/gravitino/pull/6071#issuecomment-2568562956 > By "vending", I think we mean "vendoring" ... `credential vending` is a specific word, please refer to https://docs.aws.amazon.com/lake-formation/latest/dg/aws-lake-formation-api-credential-vending.html or https://docs.databricks.com/en/external-access/credential-vending.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
tengqm commented on PR #6071: URL: https://github.com/apache/gravitino/pull/6071#issuecomment-2568550129 By "vending", I think we mean "vendoring" ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
tengqm commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1901382792 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
orenccl commented on code in PR #6071: URL: https://github.com/apache/gravitino/pull/6071#discussion_r1900962135 ## docs/security/credential-vending.md: ## @@ -0,0 +1,175 @@ +--- +title: "Gravitino credential vending" +slug: /security/credential-vending +keyword: security credential vending +license: "This software is licensed under the Apache License version 2." +--- + +## Background + +Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms. + +### Capabilities + +- Supports Gravitino Iceberg REST server. +- Supports Gravitino server, only support Hadoop catalog. +- Doesn't support Spark/Trino/Flink connector. +- With Build-in credentials: + - S3: `S3TokenCredential`, `S3SecretKeyCredential` + - GCS: `GCSTokenCredential` + - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential` + - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential` + +## General configurations + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-|||---|--|--| +| `credential-provider-type` | `gravitino.iceberg-rest.credential-provider-type` | Deprecated, please use `credential-providers` instead. | (none) | Yes | 0.7.0-incubating | +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | The credential provider types, separated by comma. | (none)| Yes | 0.8.0-incubating | +| `credential-cache-expire-ratio` | `gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the credential's expiration time when Gravitino remove credential from the cache. | 0.15 | No | 0.8.0-incubating | +| `credential-cache-max-size` | `gravitino.iceberg-rest.cache-max-size`| Max size for the credential cache. | 1 | No | 0.8.0-incubating | + +## Build-in credentials configurations + +### S3 credentials + + S3 secret key credential + +A credential with static S3 access key id and secret access key. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description| Default value | Required | Since Version| +|-|---||---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3 secret key credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data. | (none)| Yes | 0.6.0-incubating | +| `s3-secret-access-key` | `gravitino.iceberg-rest.s3-secret-access-key` | The static secret access key used to access S3 data. | (none)| Yes | 0.6.0-incubating | + + S3 token credential + +A credential with a subscope privileged token, to use S3 token credential, you should create a role and grant proper privilege to the role to access the S3 data. + +| Gravitino server catalog properties | Gravitino Iceberg REST server configurations | Description | Default value | Required | Since Version| +|-||-|---|--|--| +| `credential-providers` | `gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token credential provider. | (none)| Yes | 0.7.0-incubating | +| `s3-access-key-id` | `gravitino.iceberg-rest.s3-access-key-id` | The static access key ID used to access S3 data.
Re: [PR] [#6070][#5649] docs(core): add credential vending document [gravitino]
FANNG1 commented on PR #6071: URL: https://github.com/apache/gravitino/pull/6071#issuecomment-2567842097 @yuqi1129 @jerryshao @orenccl @sunxiaojian PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
