[
https://issues.apache.org/jira/browse/FLINK-29754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623704#comment-17623704
]
Peter Vary commented on FLINK-29754:
------------------------------------
I create a configuration file like this:
{code:java}
<?xml version="1.0"?>
<!-- core-site.xml -->
<configuration>
<property>
<name>hadoop.fs.s3a.buffer.dir</name>
<value>/flink-data</value>
</property>
<property>
<name>fs.s3a.bucket.probe</name>
<value>0</value>
</property>
[..]
</configuration> {code}
The configuration values set in the configuration files are available, and used
when accessing S3 in the case when this configuration file is on the classpath
(packaged in the jar). OTOH, if I create a HADOO_CONF_DIR and put the config
files there, then the configuration values are not available and not used when
accessing S3.
> HadoopConfigLoader should consider Hadoop configuration files
> -------------------------------------------------------------
>
> Key: FLINK-29754
> URL: https://issues.apache.org/jira/browse/FLINK-29754
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Peter Vary
> Priority: Major
>
> Currently
> [HadoopConfigLoader|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java]
> considers Hadoop configurations on the classpath, but does not consider
> Hadoop configuration files which are set in another way.
> So if the Hadoop configuration is set through the {{HADOOP_CONF_DIR}}
> environment variable, then the configuration loaded by the HadoopConfigLoader
> will not contain the values set there.
> This can cause unexpected behaviour when setting checkpoint / savepoint dirs
> on S3, and the specific S3 configurations are set in the Hadoop configuration
> files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
