I think I finally figured this out and also posted some of my findings on https://issues.apache.org/jira/browse/BEAM-2457 <https://issues.apache.org/jira/browse/BEAM-2457>.
It seems that the Hadoop config directory needs to be both on the classpath and be exported via HADOOP_CONF_DIR. This is true both when submitting the program and also on the (Flink) cluster that is running the program. For the concrete problem at hand this means: 1) The Flink cluster needs to be started like this export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_CLASSPATH=/etc/hadoop/conf bin/start-cluster.sh 2) The job needs to be submitted like this: export HADOOP_CONF_DIR=/etc/hadoop/conf java -cp word-count-beam-bundled-0.1.jar:/home/cloudera/flink-1.3.2/lib/flink-dist_2.10-1.3.2.jar org.apache.beam.examples.WordCount --runner=FlinkRunner --flinkMaster=localhost:6123 --filesToStage=/home/cloudera/word-count-beam/target/word-count-beam-bundled-0.1.jar --inputFile=hdfs:///user/aljoscha/wc-in --output=hdfs:///user/aljoscha/wc-out-13 Of course, setting it via exporting an environment variable is only the quickest way to do it. It should normally be configured in the Flink configuration. The reason why we currently need that extra flink-dist_2.10-1.3.2.jar on the classpath is that the Flink Client uses Akka and the Akka default settings are apparently stored as resources in that jar. Best, Aljoscha > On 29. Aug 2017, at 10:27, Jean-Baptiste Onofré <[email protected]> wrote: > > By the way, this kind of question should go on the user mailing list IMHO. > > Thanks > Regards > JB > > On 08/29/2017 08:59 AM, P. Ramanjaneya Reddy wrote: >> Hi All, >> build jar file from the beam quickstart. while run the jar on Flinkcluster >> got below error.? >> anybody got this error? >> Could you please help how to resolve this? >> root1@master:~/NAI/Tools/flink-1.3.0$ *bin/flink run -c >> org.apache.beam.examples.WordCount >> /home/root1/NAI/Tools/word-count-beam/target/word-count-beam-bundled-0.1.jar >> --runner=FlinkRunner >> --filesToStage=/home/root1/NAI/Tools/word-count-beam/target/word-count-beam-bundled-0.1.jar >> --inputFile=hdfs://master:9000/test/wordcount_input.txt >> --output=hdfs://master:9000/test/wordcount_output919* >> This is the output I get: >> Caused by: java.lang.IllegalStateException: Unable to find registrar for >> hdfs >> at >> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:447) >> at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:517) >> at >> org.apache.beam.sdk.io.FileBasedSink.convertToFileResourceIfPossible(FileBasedSink.java:204) >> at org.apache.beam.sdk.io.TextIO$Write.to(TextIO.java:296) >> at org.apache.beam.examples.WordCount.main(WordCount.java:182) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) >> ... 13 more >> Thanks & Regards, >> Ramanji. > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com
