Re: question on flink-storm-examples

2015-09-03 Thread Matthias J. Sax
One more remark that just came to my mind. There is a storm-hdfs module available: https://github.com/apache/storm/tree/master/external/storm-hdfs Maybe you can use it. It would be great if you could give feedback if this works for you. -Matthias On 09/02/2015 10:52 AM, Matthias J. Sax wrote: >

Re: question on flink-storm-examples

2015-09-02 Thread Matthias J. Sax
Hi, StormFileSpout uses a simple FileReader internally an cannot deal with HDFS. It would be a nice extension to have. I just opened a JIRA for it: https://issues.apache.org/jira/browse/FLINK-2606 Jerry, feel to work in this feature and contribute code to Flink ;) -Matthias On 09/02/2015 07:52

question on flink-storm-examples

2015-09-01 Thread Jerry Peng
Hello, I have some questions regarding how to run one of the flink-storm-examples, the WordCountTopology. How should I run the job? On github its says I should just execute bin/flink run example.jar but when I execute: bin/flink run WordCount-StormTopology.jar nothing happens. What am I

Re: question on flink-storm-examples

2015-09-01 Thread Jerry Peng
When I run WordCount-StormTopology I get the following exception: ~/flink/bin/flink run WordCount-StormTopology.jar hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/data.txt hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/results.txt org.apache.flink.client.program.ProgramInvocationException: The

Re: question on flink-storm-examples

2015-09-01 Thread Jerry Peng
Ya that what I did and everything seems execute fine but when I try to run the WordCount-StormTopology with a file on hfs I get a java.io.FileNotFoundException : java.lang.RuntimeException: java.io.FileNotFoundException: /home/jerrypeng/hadoop/hadoop_dir/data/data.txt (No such file or directory)

Re: question on flink-storm-examples

2015-09-01 Thread Matthias J. Sax
Hi Jerry, WordCount-StormTopology uses a hard coded dop of 4. If you start up Flink in local mode (bin/start-local-streaming.sh), you need to increase the number of task slots to at least 4 in conf/flink-conf.yaml before starting Flink -> taskmanager.numberOfTaskSlots You should actually see the

Re: question on flink-storm-examples

2015-09-01 Thread Matthias J. Sax
You can use "bin/flink cancel JOBID" or JobManager WebUI to cancel the running job. The exception you see, occurs in FlinkSubmitter.killTopology(...) which is not used by "bin/flink cancel" or JobMaanger WebUI. If you compile the example you yourself, just remove the call to killTopology().

Re: question on flink-storm-examples

2015-09-01 Thread Matthias J. Sax
Oh yes. I forgot about this. I have already a fix for it in a pending pull request... I hope that this PR is merged soon... If you want to observe the progress, look here: https://issues.apache.org/jira/browse/FLINK-2111 and https://issues.apache.org/jira/browse/FLINK-2338 This PR, resolves both

Re: question on flink-storm-examples

2015-09-01 Thread Jerry Peng
Hello, I corrected the number of slots for each task manager but now when I try to run the WordCount-StormTopology, the job manager daemon on my master node crashes and I get this exception in the log: java.lang.Exception: Received a message CancelJob(6a4b9aa01ec87db20060210e5b36065e) without a

Re: question on flink-storm-examples

2015-09-01 Thread Matthias J. Sax
Yes. That is what I expected. JobManager cannot start the job, due to less task slots. It logs the exception NoResourceAvailableException (it is not shown in stdout; see "log" folder). There is no feedback to Flink CLI that the job could not be started. Furthermore, WordCount-StormTopology