Hi,
Flink is not loading the Hadoop configuration from the classloader. You
have to specify the path to the Hadoop configuration in the flink
configuration "fs.hdfs.hadoopconf"
On Thu, Jun 25, 2015 at 2:50 PM, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> Hi to all,
> I'm experiencing some problem in writing a file as csv on HDFS with flink
> 0.9.0.
> The code I use is
> myDataset.writeAsCsv(new Path("hdfs:///tmp", "myFile.csv").toString());
>
> If I run the job from Eclipse everything works fine but when I deploy the
> job on the cluster (cloudera 5.1.3) I obtain the following exception:
>
> Caused by: java.io.IOException: The given HDFS file URI
> (hdfs:///tmp/myFile.csv) did not describe the HDFS NameNode. The attempt to
> use a default HDFS configuration, as specified in the 'fs.hdfs.hdfsdefault'
> or 'fs.hdfs.hdfssite' config parameter failed due to the following problem:
> Either no default file system was registered, or the provided configuration
> contains no valid authority component (fs.default.name or fs.defaultFS)
> describing the (hdfs namenode) host and port.
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:291)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:258)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:309)
> at
> org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:273)
> at
> org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:84)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$4.apply(JobManager.scala:520)
> ... 25 more
>
> The core-site.xml is present in the fat jar and contains the property
>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://myServerX:8020</value>
> </property>
>
> I compiled flink with the following command:
>
> mvn clean install -Dhadoop.version=2.3.0-cdh5.1.3
> -Dhbase.version=0.98.1-cdh5.1.3 -Dhadoop.core.version=2.3.0-mr1-cdh5.1.3
> -DskipTests -Pvendor-repos
>
> How can I fix that?
>
> Best,
> Flavio
>
>
