[
https://issues.apache.org/jira/browse/FLINK-9245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489033#comment-16489033
]
Julien Cuquemelle commented on FLINK-9245:
------------------------------------------
Hi,
Maybe I was not clear when describing the issue, as in fact our Urls are not
hardcoded like in my example, but are provided by other components.
we have a production system that doesn't specify authorities in its URLs
because the default Hadoop config is available in the environment.
However, we tests such systems (which should work with the same URLs) on
environments without a default hadoop conf, by providing a valid hadoop
configuration at runtime. However, Flink is not able to use this provided
configuration if a default one is not available on the environment (which
should not be necessary IMHO).
Anyway, I guess this use-case is not so common, and the focus is not currently
strong on our side, so we can keep the ticket closed. I'll reopen it if the
need for the fix becomes more important.
thanks for reviewing and your comments anyway !
> Can't create a BucketingSink with a provided Configuration if no hadoop
> defaults
> --------------------------------------------------------------------------------
>
> Key: FLINK-9245
> URL: https://issues.apache.org/jira/browse/FLINK-9245
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 1.4.2
> Reporter: Julien Cuquemelle
> Priority: Minor
>
> We build Integration tests using this kind of code:
> {code:java}
> val bucketingSink = new
> BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
> bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs)
> bucketingSink.setBucketer(new DateTimeBucketer[Row]("yyyy-MM-dd--HHmm"))
> outpuStream.addSink(bucketingSink)
> {code}
> Here, the hadoopRule is providing a valid hdfs config that should allows this
> kind of code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set
> up, like a developper workstation or a Jenkins slave.
> When running this code on such a machine, the .createHadoopFileSystem(...)
> fails with
> {noformat}
> The given file system URI (hdfs:///user/$USER/application_name/) did not
> describe the authority
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
> because it tries to instantiate the fileSystem from a default configuration
> in .getUnguardedFileSystem() ; as the default conf doesn't exist, the default
> filesystem resolves to "file:///" and the checks of the consistency of the
> URI fails because no authority can be found So the whole filesystem creation
> fails before actually trying to use the provided config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
