[
https://issues.apache.org/jira/browse/FLINK-24595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721160#comment-17721160
]
Lu commented on FLINK-24595:
----------------------------
i have the save problem in 1.15.2, it seems
flink-s3-fs-hadoop read s3 configuration from env , not from the configuration
obj
> Programmatic configuration of S3 doesn't pass parameters to Hadoop FS
> ---------------------------------------------------------------------
>
> Key: FLINK-24595
> URL: https://issues.apache.org/jira/browse/FLINK-24595
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility
> Affects Versions: 1.14.0
> Environment: Flink 1.14.0
> JDK 8
> {{openjdk version "1.8.0_302"}}
> {{OpenJDK Runtime Environment (Zulu 8.56.0.23-CA-macos-aarch64) (build
> 1.8.0_302-b08)}}
> {{OpenJDK 64-Bit Server VM (Zulu 8.56.0.23-CA-macos-aarch64) (build
> 25.302-b08, mixed mode)}}
> Reporter: Pavel Penkov
> Priority: Major
> Attachments: FlinkApp.java, TickingSource.java, flink_exception.txt
>
>
> When running in mini-cluster mode Flink apparently doesn't pass S3
> configuration to underlying Hadoop FS. With a code like this
> {code:java}
> Configuration conf = new Configuration();
> conf.setString("s3.endpoint", "http://localhost:4566";);
> conf.setString("s3.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
> conf.setString("s3.access.key", "harvester");
> conf.setString("s3.secret.key", "harvester");
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment(conf);
> {code}
> Application fails with an exception with most relevant error being {{Caused
> by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
> provided by SimpleAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider :
> com.amazonaws.SdkClientException: Failed to connect to service endpoint: }}
> So Hadoop lists all the providers but it should use only the one set in
> configuration. Full project that reproduces this behaviour is available at
> [https://github.com/PavelPenkov/flink-s3-conf] and relevant files are
> attached to this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
