Pavel Penkov created FLINK-24595: ------------------------------------ Summary: Programmatic configuration of S3 doesn't pass parameters to Hadoop FS Key: FLINK-24595 URL: https://issues.apache.org/jira/browse/FLINK-24595 Project: Flink Issue Type: Bug Components: Connectors / Hadoop Compatibility Affects Versions: 1.14.0 Environment: Flink 1.14.0
JDK 8 {{openjdk version "1.8.0_302"}} {{OpenJDK Runtime Environment (Zulu 8.56.0.23-CA-macos-aarch64) (build 1.8.0_302-b08)}} {{OpenJDK 64-Bit Server VM (Zulu 8.56.0.23-CA-macos-aarch64) (build 25.302-b08, mixed mode)}} Reporter: Pavel Penkov Attachments: FlinkApp.java, TickingSource.java, flink_exception.txt When running in mini-cluster mode Flink apparently doesn't pass S3 configuration to underlying Hadoop FS. With a code like this {code:java} Configuration conf = new Configuration(); conf.setString("s3.endpoint", "http://localhost:4566"); conf.setString("s3.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"); conf.setString("s3.access.key", "harvester"); conf.setString("s3.secret.key", "harvester"); StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(conf); {code} Application fails with an exception with most relevant error being {{Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Failed to connect to service endpoint: }} So Hadoop lists all the providers but it should use only the one set in configuration. Full project that reproduces this behaviour is available at [https://github.com/PavelPenkov/flink-s3-conf] and relevant files are attached to this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)