[jira] [Closed] (FLINK-9245) Can't create a BucketingSink with a provided Configuration if no hadoop defaults
[
https://issues.apache.org/jira/browse/FLINK-9245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-9245.
---
Resolution: Not A Problem
> Can't create a BucketingSink with a provided Configuration if no hadoop
> defaults
>
>
> Key: FLINK-9245
> URL: https://issues.apache.org/jira/browse/FLINK-9245
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.4.2
>Reporter: Julien Cuquemelle
>Priority: Minor
>
> We build Integration tests using this kind of code:
> {code:java}
> val bucketingSink = new
> BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
> bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs)
> bucketingSink.setBucketer(new DateTimeBucketer[Row]("-MM-dd--HHmm"))
> outpuStream.addSink(bucketingSink)
> {code}
> Here, the hadoopRule is providing a valid hdfs config that should allows this
> kind of code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set
> up, like a developper workstation or a Jenkins slave.
> When running this code on such a machine, the .createHadoopFileSystem(...)
> fails with
> {noformat}
> The given file system URI (hdfs:///user/$USER/application_name/) did not
> describe the authority
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
> because it tries to instantiate the fileSystem from a default configuration
> in .getUnguardedFileSystem() ; as the default conf doesn't exist, the default
> filesystem resolves to "file:///" and the checks of the consistency of the
> URI fails because no authority can be found So the whole filesystem creation
> fails before actually trying to use the provided config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-9245) Can't create a BucketingSink with a provided Configuration if no hadoop defaults
[
https://issues.apache.org/jira/browse/FLINK-9245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-9245.
---
Resolution: Not A Problem
Closing now but please reopen if specifying an authority (host) doesn't fix it.
> Can't create a BucketingSink with a provided Configuration if no hadoop
> defaults
>
>
> Key: FLINK-9245
> URL: https://issues.apache.org/jira/browse/FLINK-9245
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.4.2
>Reporter: Julien Cuquemelle
>Priority: Minor
> Fix For: 1.6.0
>
>
> We build Integration tests using this kind of code:
> {code:java}
> val bucketingSink = new
> BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
> bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs)
> bucketingSink.setBucketer(new DateTimeBucketer[Row]("-MM-dd--HHmm"))
> outpuStream.addSink(bucketingSink)
> {code}
> Here, the hadoopRule is providing a valid hdfs config that should allows this
> kind of code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set
> up, like a developper workstation or a Jenkins slave.
> When running this code on such a machine, the .createHadoopFileSystem(...)
> fails with
> {noformat}
> The given file system URI (hdfs:///user/$USER/application_name/) did not
> describe the authority
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
> because it tries to instantiate the fileSystem from a default configuration
> in .getUnguardedFileSystem() ; as the default conf doesn't exist, the default
> filesystem resolves to "file:///" and the checks of the consistency of the
> URI fails because no authority can be found So the whole filesystem creation
> fails before actually trying to use the provided config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
