This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 7a8b09002 [#5839] feat(core): support custom STS Endpoint for AWS S3
(#5801)
7a8b09002 is described below
commit 7a8b09002a7a26f95965493532b1822f9edbcdfa
Author: Hien Pham <[email protected]>
AuthorDate: Mon Dec 16 10:11:10 2024 +0700
[#5839] feat(core): support custom STS Endpoint for AWS S3 (#5801)
### What changes were proposed in this pull request?
Support config STS Endpoint for S3 in Rest Catalog
### Why are the changes needed?
Support for On-premise S3 Deployment such as MinIO
Fixe: #5839
### Does this PR introduce _any_ user-facing change?
1. Addition of property keys `s3-token-service-endpoint` for iceberg
catalog.
### How was this patch tested?
Tested locally.
---
.../gravitino/s3/credential/S3TokenProvider.java | 7 ++++++
.../org/apache/gravitino/storage/S3Properties.java | 2 ++
.../credential/config/S3CredentialConfig.java | 11 +++++++++
docs/iceberg-rest-service.md | 26 ++++++++++++----------
4 files changed, 34 insertions(+), 12 deletions(-)
diff --git
a/bundles/aws-bundle/src/main/java/org/apache/gravitino/s3/credential/S3TokenProvider.java
b/bundles/aws-bundle/src/main/java/org/apache/gravitino/s3/credential/S3TokenProvider.java
index 3c14d410c..24b88875d 100644
---
a/bundles/aws-bundle/src/main/java/org/apache/gravitino/s3/credential/S3TokenProvider.java
+++
b/bundles/aws-bundle/src/main/java/org/apache/gravitino/s3/credential/S3TokenProvider.java
@@ -104,6 +104,13 @@ public class S3TokenProvider implements CredentialProvider
{
if (StringUtils.isNotBlank(region)) {
builder.region(Region.of(region));
}
+ String stsEndpoint = s3CredentialConfig.stsEndpoint();
+ // If the user does not set a value or provides an blank string, we treat
as unspecified.
+ // The goal is to pass a custom endpoint to the `builder` only when the
user specifies a
+ // non-blank value.
+ if (StringUtils.isNotBlank(stsEndpoint)) {
+ builder.endpointOverride(URI.create(stsEndpoint));
+ }
return builder.build();
}
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/S3Properties.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/S3Properties.java
index 2dbe67649..cfb342c5b 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/S3Properties.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/S3Properties.java
@@ -32,6 +32,8 @@ public class S3Properties {
public static final String GRAVITINO_S3_REGION = "s3-region";
// S3 role arn
public static final String GRAVITINO_S3_ROLE_ARN = "s3-role-arn";
+
+ public static final String GRAVITINO_S3_STS_ENDPOINT =
"s3-token-service-endpoint";
// S3 external id
public static final String GRAVITINO_S3_EXTERNAL_ID = "s3-external-id";
diff --git
a/core/src/main/java/org/apache/gravitino/credential/config/S3CredentialConfig.java
b/core/src/main/java/org/apache/gravitino/credential/config/S3CredentialConfig.java
index 7f282d7e4..1bdf7b2fa 100644
---
a/core/src/main/java/org/apache/gravitino/credential/config/S3CredentialConfig.java
+++
b/core/src/main/java/org/apache/gravitino/credential/config/S3CredentialConfig.java
@@ -76,6 +76,13 @@ public class S3CredentialConfig extends Config {
.intConf()
.createWithDefault(3600);
+ public static final ConfigEntry<String> S3_STS_ENDPOINT =
+ new ConfigBuilder(S3Properties.GRAVITINO_S3_STS_ENDPOINT)
+ .doc("S3 STS endpoint")
+ .version(ConfigConstants.VERSION_0_8_0)
+ .stringConf()
+ .create();
+
public S3CredentialConfig(Map<String, String> properties) {
super(false);
loadFromMap(properties, k -> true);
@@ -107,4 +114,8 @@ public class S3CredentialConfig extends Config {
public Integer tokenExpireInSecs() {
return this.get(S3_TOKEN_EXPIRE_IN_SECS);
}
+
+ public String stsEndpoint() {
+ return this.get(S3_STS_ENDPOINT);
+ }
}
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index 733ace659..862bb0486 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -117,6 +117,7 @@ Gravitino Iceberg REST service supports using static S3
secret key or generating
| `gravitino.iceberg-rest.s3-role-arn` | The ARN of the role to
access the S3 data.
| (none) | Yes, when
`credential-provider-type` is `s3-token` | 0.7.0-incubating |
| `gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token, only used when `credential-provider-type` is `s3-token`.
| (none) | No
| 0.7.0-incubating |
| `gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role, only used when `credential-provider-type` is `s3-token`.
| 3600 | No
| 0.7.0-incubating |
+| `gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative
endpoint of the S3 token service, This could be used with s3-compatible object
storage service like MINIO that has a different STS endpoint. | (none) |
No | 0.8.0-incubating |
For other Iceberg s3 properties not managed by Gravitino like `s3.sse.type`,
you could config it directly by `gravitino.iceberg-rest.s3.sse.type`.
@@ -417,18 +418,19 @@ docker run -d -p 9001:9001
apache/gravitino-iceberg-rest:0.7.0-incubating
Gravitino Iceberg REST server in docker image could access local storage by
default, you could set the following environment variables if the storage is
cloud/remote storage like S3, please refer to [storage section](#storage) for
more details.
-| Environment variables | Configuration items
| Since version |
-|--------------------------------------|---------------------------------------------------|-------------------|
-| `GRAVITINO_IO_IMPL` | `gravitino.iceberg-rest.io-impl`
| 0.7.0-incubating |
-| `GRAVITINO_URI` | `gravitino.iceberg-rest.uri`
| 0.7.0-incubating |
-| `GRAVITINO_WAREHOUSE` | `gravitino.iceberg-rest.warehouse`
| 0.7.0-incubating |
-| `GRAVITINO_CREDENTIAL_PROVIDER_TYPE` |
`gravitino.iceberg-rest.credential-provider-type` | 0.7.0-incubating |
-| `GRAVITINO_GCS_CREDENTIAL_FILE_PATH` |
`gravitino.iceberg-rest.gcs-credential-file-path` | 0.7.0-incubating |
-| `GRAVITINO_S3_ACCESS_KEY` |
`gravitino.iceberg-rest.s3-access-key-id` | 0.7.0-incubating |
-| `GRAVITINO_S3_SECRET_KEY` |
`gravitino.iceberg-rest.s3-secret-access-key` | 0.7.0-incubating |
-| `GRAVITINO_S3_REGION` | `gravitino.iceberg-rest.s3-region`
| 0.7.0-incubating |
-| `GRAVITINO_S3_ROLE_ARN` | `gravitino.iceberg-rest.s3-role-arn`
| 0.7.0-incubating |
-| `GRAVITINO_S3_EXTERNAL_ID` |
`gravitino.iceberg-rest.s3-external-id` | 0.7.0-incubating |
+| Environment variables | Configuration items
| Since version |
+|------------------------------------------------|---------------------------------------------------|-------------------|
+| `GRAVITINO_IO_IMPL` |
`gravitino.iceberg-rest.io-impl` | 0.7.0-incubating |
+| `GRAVITINO_URI` |
`gravitino.iceberg-rest.uri` | 0.7.0-incubating |
+| `GRAVITINO_WAREHOUSE` |
`gravitino.iceberg-rest.warehouse` | 0.7.0-incubating |
+| `GRAVITINO_CREDENTIAL_PROVIDER_TYPE` |
`gravitino.iceberg-rest.credential-provider-type` | 0.7.0-incubating |
+| `GRAVITINO_GCS_CREDENTIAL_FILE_PATH` |
`gravitino.iceberg-rest.gcs-credential-file-path` | 0.7.0-incubating |
+| `GRAVITINO_S3_ACCESS_KEY` |
`gravitino.iceberg-rest.s3-access-key-id` | 0.7.0-incubating |
+| `GRAVITINO_S3_SECRET_KEY` |
`gravitino.iceberg-rest.s3-secret-access-key` | 0.7.0-incubating |
+| `GRAVITINO_S3_REGION` |
`gravitino.iceberg-rest.s3-region` | 0.7.0-incubating |
+| `GRAVITINO_S3_ROLE_ARN` |
`gravitino.iceberg-rest.s3-role-arn` | 0.7.0-incubating |
+| `GRAVITINO_S3_EXTERNAL_ID` |
`gravitino.iceberg-rest.s3-external-id` | 0.7.0-incubating |
+| `GRAVITINO_S3_TOKEN_SERVICE_ENDPOINT` |
`gravitino.iceberg-rest.s3-token-service-endpoint`| 0.8.0-incubating |
Or build it manually to add custom configuration or logics:
