One more remark that just came to my mind. There is a storm-hdfs module
available: https://github.com/apache/storm/tree/master/external/storm-hdfs
Maybe you can use it. It would be great if you could give feedback if
this works for you.
-Matthias
On 09/02/2015 10:52 AM, Matthias J. Sax wrote:
>
Hi,
StormFileSpout uses a simple FileReader internally an cannot deal with
HDFS. It would be a nice extension to have. I just opened a JIRA for it:
https://issues.apache.org/jira/browse/FLINK-2606
Jerry, feel to work in this feature and contribute code to Flink ;)
-Matthias
On 09/02/2015 07:52
Hello,
I have some questions regarding how to run one of the flink-storm-examples,
the WordCountTopology. How should I run the job? On github its says I
should just execute
bin/flink run example.jar but when I execute:
bin/flink run WordCount-StormTopology.jar
nothing happens. What am I
When I run WordCount-StormTopology I get the following exception:
~/flink/bin/flink run WordCount-StormTopology.jar
hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/data.txt
hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/results.txt
org.apache.flink.client.program.ProgramInvocationException: The
Ya that what I did and everything seems execute fine but when I try to run
the WordCount-StormTopology with a file on hfs I get a
java.io.FileNotFoundException
:
java.lang.RuntimeException: java.io.FileNotFoundException:
/home/jerrypeng/hadoop/hadoop_dir/data/data.txt (No such file or directory)
Hi Jerry,
WordCount-StormTopology uses a hard coded dop of 4. If you start up
Flink in local mode (bin/start-local-streaming.sh), you need to increase
the number of task slots to at least 4 in conf/flink-conf.yaml before
starting Flink -> taskmanager.numberOfTaskSlots
You should actually see the
You can use "bin/flink cancel JOBID" or JobManager WebUI to cancel the
running job.
The exception you see, occurs in FlinkSubmitter.killTopology(...) which
is not used by "bin/flink cancel" or JobMaanger WebUI.
If you compile the example you yourself, just remove the call to
killTopology().
Oh yes. I forgot about this. I have already a fix for it in a pending
pull request... I hope that this PR is merged soon...
If you want to observe the progress, look here:
https://issues.apache.org/jira/browse/FLINK-2111
and
https://issues.apache.org/jira/browse/FLINK-2338
This PR, resolves both
Hello,
I corrected the number of slots for each task manager but now when I try to
run the WordCount-StormTopology, the job manager daemon on my master node
crashes and I get this exception in the log:
java.lang.Exception: Received a message
CancelJob(6a4b9aa01ec87db20060210e5b36065e) without a
Yes. That is what I expected.
JobManager cannot start the job, due to less task slots. It logs the
exception NoResourceAvailableException (it is not shown in stdout; see
"log" folder). There is no feedback to Flink CLI that the job could not
be started.
Furthermore, WordCount-StormTopology
10 matches
Mail list logo