Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
affo commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2871191374
##
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme,
Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}",
accessKey);
-AWSSecurityTokenService stsClient =
-AWSSecurityTokenServiceClientBuilder.standard()
-.withRegion(region)
-.withCredentials(
-new AWSStaticCredentialsProvider(
-new BasicAWSCredentials(accessKey,
secretKey)))
+StsClient stsClient =
Review Comment:
@luoyuxia I plan to restructure this part of the code by addressing
https://github.com/apache/fluss/issues/2662 anyhow 🤝
The idea is to allow more flexible `AWSCredentialsProviders` to delegate
tokens to clients.
Would probably fix under some peculiar configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
luoyuxia commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2870239144
##
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme,
Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}",
accessKey);
-AWSSecurityTokenService stsClient =
-AWSSecurityTokenServiceClientBuilder.standard()
-.withRegion(region)
-.withCredentials(
-new AWSStaticCredentialsProvider(
-new BasicAWSCredentials(accessKey,
secretKey)))
+StsClient stsClient =
Review Comment:
Actaully, it does't matter if it don't support GetSessionToken. We can still
just use STS tokens just like we did for oss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on PR #2674: URL: https://github.com/apache/fluss/pull/2674#issuecomment-3949981803 I see Flink is also working on a similar issue: https://github.com/apache/flink/pull/27026. Let's wait and see how it progresses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2836010058 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,33 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 Review Comment: maybe we can use localstack to replace rustfs?: https://docs.localstack.cloud/aws/services/sts/#api-coverage it supports GetSessionToken API which is precisely what our code attempts to invoke @luoyuxia @wuchong -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2836006840 ## fluss-filesystems/fluss-fs-gs/src/main/resources/META-INF/NOTICE: ## @@ -75,21 +85,25 @@ This project bundles the following dependencies under the Apache Software Licens - io.opencensus:opencensus-impl-core:0.31.0 - io.opencensus:opencensus-proto:0.2.0 - io.perfmark:perfmark-api:0.26.0 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 - org.apache.httpcomponents:httpclient:4.5.13 - org.apache.httpcomponents:httpcore:4.4.13 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: After investigating the transitive dependency tree, here's where each of the new dependencies comes from: - netty: new in hadoop 3.4. Hadoop's netty usage is for internal RPC/transport, not needed for filesystem operations. - io.dropwizard.metrics: new in hadoop 3.4. Hadoop metrics framework, not needed for filesystem operations. - org.codehaus.jettison: new in hadoop 3.4. Not needed. - org.reactivestream: from AWS SDK v2 http-client-spi. It is core part of AWS SDK v2's async model, cannot be exclude. By the way, the size of the fluss-fs-s3 jar is only ~40MB. Not too much difference. @wuchong -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
affo commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2811241841 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 +- io.netty:netty-buffer:4.1.100.Final +- io.netty:netty-codec:4.1.100.Final +- io.netty:netty-common:4.1.100.Final +- io.netty:netty-handler:4.1.100.Final +- io.netty:netty-resolver:4.1.100.Final +- io.netty:netty-transport:4.1.100.Final +- io.netty:netty-transport-classes-epoll:4.1.100.Final +- io.netty:netty-transport-native-epoll:4.1.100.Final +- io.netty:netty-transport-native-unix-common:4.1.100.Final +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: @xx789633 well, that is an investigation, thank you 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
wuchong commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2810475733
##
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##
@@ -9,28 +9,33 @@ This project bundles the following dependencies under the
Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
Review Comment:
`io.dropwizard.metrics:metrics-core` is not used by `hadoop-common`
(https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/dependency-analysis.html),
can you remove it from all the NOTICE files in filesystem modules?
##
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme,
Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}",
accessKey);
-AWSSecurityTokenService stsClient =
-AWSSecurityTokenServiceClientBuilder.standard()
-.withRegion(region)
-.withCredentials(
-new AWSStaticCredentialsProvider(
-new BasicAWSCredentials(accessKey,
secretKey)))
+StsClient stsClient =
Review Comment:
@luoyuxia , do you have other solutions to resolve the rustfs issue?
##
fluss-filesystems/fluss-fs-gs/src/main/resources/META-INF/NOTICE:
##
@@ -75,21 +85,25 @@ This project bundles the following dependencies under the
Apache Software Licens
- io.opencensus:opencensus-impl-core:0.31.0
- io.opencensus:opencensus-proto:0.2.0
- io.perfmark:perfmark-api:0.26.0
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.13
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
`org.codehaus.jettison:jettison` is not used by `hadoop-common`
(https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/dependency-analysis.html),
can you remove it from all the NOTICE files in filesystem modules?
##
fluss-filesystems/fluss-fs-s3/src/main/resources/META-INF/NOTICE:
##
@@ -3,57 +3,96 @@ Copyright 2025-2026 The Apache Software Foundation
This project bundles the following dependencies under the Apache Software
License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
-- com.amazonaws:aws-java-sdk-core:1.12.319
-- com.amazonaws:aws-java-sdk-dynamodb:1.12.319
-- com.amazonaws:aws-java-sdk-kms:1.12.319
-- com.amazonaws:aws-java-sdk-s3:1.12.319
-- com.amazonaws:aws-java-sdk-sts:1.12.319
-- com.amazonaws:jmespath-java:1.12.319
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+-
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2807381847 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 +- io.netty:netty-buffer:4.1.100.Final +- io.netty:netty-codec:4.1.100.Final +- io.netty:netty-common:4.1.100.Final +- io.netty:netty-handler:4.1.100.Final +- io.netty:netty-resolver:4.1.100.Final +- io.netty:netty-transport:4.1.100.Final +- io.netty:netty-transport-classes-epoll:4.1.100.Final +- io.netty:netty-transport-native-epoll:4.1.100.Final +- io.netty:netty-transport-native-unix-common:4.1.100.Final +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: Hi @wuchong @affo, I have excluded netty dependency. netty dependency is introduced by Hadoop-common 3.4. I have checked the source code of hadoop-common 3.4 and found it does not use netty at all. The Hadoop community likely declared these dependencies in hadoop-common to unify Netty version management across all sub-modules (such as hadoop-hdfs, hadoop-yarn, and other server-side components), but hadoop-common itself does not actually use them. By the way, netty in Hadoop is mainly used for IPC/RPC, while Fluss only calls standard filesystem calls. For kerby, those dependencies were there before this pull request. We just upgrade their versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
affo commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2804027113 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 +- io.netty:netty-buffer:4.1.100.Final +- io.netty:netty-codec:4.1.100.Final +- io.netty:netty-common:4.1.100.Final +- io.netty:netty-handler:4.1.100.Final +- io.netty:netty-resolver:4.1.100.Final +- io.netty:netty-transport:4.1.100.Final +- io.netty:netty-transport-classes-epoll:4.1.100.Final +- io.netty:netty-transport-native-epoll:4.1.100.Final +- io.netty:netty-transport-native-unix-common:4.1.100.Final +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: @xx789633 these new ones are introduced by Hadoop 3.4 right? If we set them as exclusions, would everything fall apart? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2803114355
##
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme,
Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}",
accessKey);
-AWSSecurityTokenService stsClient =
-AWSSecurityTokenServiceClientBuilder.standard()
-.withRegion(region)
-.withCredentials(
-new AWSStaticCredentialsProvider(
-new BasicAWSCredentials(accessKey,
secretKey)))
+StsClient stsClient =
Review Comment:
I don't think so. According to the rustfs discussion at
https://github.com/orgs/rustfs/discussions/1746, rustfs currently only support
STS AssumeRole action and does not yet support GetSessionToken, which is
precisely what our code attempts to invoke
(https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java#L80)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
luoyuxia commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2802937454
##
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme,
Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}",
accessKey);
-AWSSecurityTokenService stsClient =
-AWSSecurityTokenServiceClientBuilder.standard()
-.withRegion(region)
-.withCredentials(
-new AWSStaticCredentialsProvider(
-new BasicAWSCredentials(accessKey,
secretKey)))
+StsClient stsClient =
Review Comment:
Seems this will also fix #2661 ? Right?
Could you please help verify it?
Also check minio can be supported or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2802937215 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 +- io.netty:netty-buffer:4.1.100.Final +- io.netty:netty-codec:4.1.100.Final +- io.netty:netty-common:4.1.100.Final +- io.netty:netty-handler:4.1.100.Final +- io.netty:netty-resolver:4.1.100.Final +- io.netty:netty-transport:4.1.100.Final +- io.netty:netty-transport-classes-epoll:4.1.100.Final +- io.netty:netty-transport-native-epoll:4.1.100.Final +- io.netty:netty-transport-native-unix-common:4.1.100.Final +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: it seems those dependencies are always included somewhere (like in https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-azure/src/main/resources/META-INF/NOTICE) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
wuchong commented on code in PR #2674: URL: https://github.com/apache/fluss/pull/2674#discussion_r2802891680 ## fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE: ## @@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-annotations:2.15.3 - com.fasterxml.jackson.core:jackson-core:2.15.3 - com.fasterxml.jackson.core:jackson-databind:2.15.3 -- com.fasterxml.woodstox:woodstox-core:5.3.0 +- com.fasterxml.woodstox:woodstox-core:5.4.0 - com.google.guava:failureaccess:1.0 - com.google.guava:guava:27.0-jre - com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava - com.google.j2objc:j2objc-annotations:1.1 - commons-beanutils:commons-beanutils:1.9.4 - commons-collections:commons-collections:3.2.2 -- commons-io:commons-io:2.8.0 -- commons-logging:commons-logging:1.1.3 -- org.apache.commons:commons-compress:1.21 -- org.apache.commons:commons-configuration2:2.1.1 +- commons-io:commons-io:2.16.1 +- commons-logging:commons-logging:1.2 +- io.dropwizard.metrics:metrics-core:3.2.4 +- io.netty:netty-buffer:4.1.100.Final +- io.netty:netty-codec:4.1.100.Final +- io.netty:netty-common:4.1.100.Final +- io.netty:netty-handler:4.1.100.Final +- io.netty:netty-resolver:4.1.100.Final +- io.netty:netty-transport:4.1.100.Final +- io.netty:netty-transport-classes-epoll:4.1.100.Final +- io.netty:netty-transport-native-epoll:4.1.100.Final +- io.netty:netty-transport-native-unix-common:4.1.100.Final +- org.apache.commons:commons-compress:1.26.1 +- org.apache.commons:commons-configuration2:2.10.1 - org.apache.commons:commons-lang3:3.18.0 -- org.apache.commons:commons-text:1.4 -- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 -- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 -- org.apache.hadoop:hadoop-annotations:3.3.4 -- org.apache.hadoop:hadoop-auth:3.3.4 -- org.apache.hadoop:hadoop-common:3.3.4 -- org.apache.kerby:kerb-core:1.0.1 -- org.apache.kerby:kerby-asn1:1.0.1 -- org.apache.kerby:kerby-pkix:1.0.1 -- org.apache.kerby:kerby-util:1.0.1 +- org.apache.commons:commons-text:1.10.0 +- org.apache.hadoop:hadoop-annotations:3.4.1 +- org.apache.hadoop:hadoop-auth:3.4.1 +- org.apache.hadoop:hadoop-common:3.4.1 +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 +- org.apache.kerby:kerb-core:2.0.3 +- org.apache.kerby:kerb-crypto:2.0.3 +- org.apache.kerby:kerb-util:2.0.3 +- org.apache.kerby:kerby-asn1:2.0.3 +- org.apache.kerby:kerby-config:2.0.3 +- org.apache.kerby:kerby-pkix:2.0.3 +- org.apache.kerby:kerby-util:2.0.3 +- org.codehaus.jettison:jettison:1.5.4 Review Comment: Are the new dependencies really needed? Especially the netty and kerby dependecies. Netty is a big dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
