Basically I have added input and outputfiles to HDFS.

But how to specify access the input file in commandline to run wordcount
program?

bin/flink run ./examples/streaming/WordCount.jar --input
*hdfs:///user/wordcount_input.txt* --output
*hdfs:///user/wordcount_output.txt *

On Tue, Aug 8, 2017 at 3:48 PM, P. Ramanjaneya Reddy <ramanji...@gmail.com>
wrote:

> Yes Felix,
>
> I have created input and output files in HDFS.
> http://localhost:50070/explorer.html#/user
>
>
> But how we can access it ?
>
> bin/flink run ./examples/streaming/WordCount.jar --input
> *hdfs:///user/wordcount_input.txt* --output
> *hdfs:///user/wordcount_output.txt *
>
>
>
> On Tue, Aug 8, 2017 at 2:55 PM, Felix Neutatz <neut...@googlemail.com>
> wrote:
>
>> Hi,
>>
>> like Timo said e.g. you need a distributed filesystem like HDFS.
>>
>> Best regards,
>> Felix
>>
>> On Aug 8, 2017 09:01, "P. Ramanjaneya Reddy" <ramanji...@gmail.com>
>> wrote:
>>
>> Hi Timo,
>>
>> How to make access the files across TM?
>>
>> Thanks & Regards,
>> Ramanji.
>>
>> On Mon, Aug 7, 2017 at 7:45 PM, Timo Walther <twal...@apache.org> wrote:
>>
>> > Flink is a distributed software for clusters. You need something like a
>> > distributed file system. So that input file and output files can be
>> > accessed from all nodes.
>> >
>> > Each TM has a log directory where the execution logs are stored.
>> >
>> > You can set additional properties to your output format by importing the
>> > code in your IDE.
>> >
>> > Am 07.08.17 um 16:03 schrieb P. Ramanjaneya Reddy:
>> >
>> > Hi Timo,
>> >> Problem is resolved after copy input file to all tasks managers.
>> >>
>> >> and where should generate outputfile? Is it in jobmanager or task
>> manager?
>> >>
>> >> Where can i see the execution logs to understand how word count done
>> each
>> >> task manager?
>> >>
>> >>
>> >> By the way any option to overwride...?
>> >>
>> >> 08/07/2017 19:27:00 Keyed Aggregation -> Sink: Unnamed(1/1) switched to
>> >> FAILED
>> >> java.io.IOException: File or directory already exists. Existing files
>> and
>> >> directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE
>> mode
>> >> to
>> >> overwrite existing files and directories.
>> >> at
>> >> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileS
>> >> ystem.java:763)
>> >> at
>> >> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.initOutP
>> >> athLocalFS(SafetyNetWrapperFileSystem.java:135)
>> >> at
>> >> org.apache.flink.api.common.io.FileOutputFormat.open(FileOut
>> >> putFormat.java:231)
>> >> at
>> >> org.apache.flink.api.java.io.TextOutputFormat.open(TextOutpu
>> >> tFormat.java:78)
>> >> at
>> >> org.apache.flink.streaming.api.functions.sink.OutputFormatSi
>> >> nkFunction.open(OutputFormatSinkFunction.java:61)
>> >> at
>> >> org.apache.flink.api.common.functions.util.FunctionUtils.ope
>> >> nFunction(FunctionUtils.java:36)
>> >> at
>> >> org.apache.flink.streaming.api.operators.AbstractUdfStreamOp
>> >> erator.open(AbstractUdfStreamOperator.java:111)
>> >> at
>> >> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO
>> >> perators(StreamTask.java:376)
>> >> at
>> >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>> >> StreamTask.java:253)
>> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >>
>> >>
>> >> On Mon, Aug 7, 2017 at 6:49 PM, Timo Walther <twal...@apache.org>
>> wrote:
>> >>
>> >> Make sure that the file exists and is accessible from all Flink tasks
>> >>> managers.
>> >>>
>> >>>
>> >>> Am 07.08.17 um 14:35 schrieb P. Ramanjaneya Reddy:
>> >>>
>> >>> Thank you Timo.
>> >>>>
>> >>>>
>> >>>> root1@root1-HP-EliteBook-840-G2:~/NAI/Tools/BEAM/Flink_Clust
>> >>>> er/rama/flink$
>> >>>> *./bin/flink
>> >>>> run ./examples/streaming/WordCount.jar --input
>> >>>> file:///home/root1/hamlet.txt --output file:///home/root1/wordcount_o
>> >>>> ut*
>> >>>>
>> >>>>
>> >>>>
>> >>>> Execution of worcountjar gives error...
>> >>>>
>> >>>> 08/07/2017 18:03:16 Source: Custom File Source(1/1) switched to
>> FAILED
>> >>>> java.io.FileNotFoundException: The provided file path
>> >>>> file:/home/root1/hamlet.txt does not exist.
>> >>>> at
>> >>>> org.apache.flink.streaming.api.functions.source.ContinuousFi
>> >>>> leMonitoringFunction.run(ContinuousFileMonitoringFunction.java:192)
>> >>>> at
>> >>>> org.apache.flink.streaming.api.operators.StreamSource.run(
>> >>>> StreamSource.java:87)
>> >>>> at
>> >>>> org.apache.flink.streaming.api.operators.StreamSource.run(
>> >>>> StreamSource.java:55)
>> >>>> at
>> >>>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.
>> >>>> run(SourceStreamTask.java:95)
>> >>>> at
>> >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>> >>>> StreamTask.java:263)
>> >>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>> >>>> at java.lang.Thread.run(Thread.java:748)
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 7, 2017 at 5:56 PM, Timo Walther <twal...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>> Hi Ramanji,
>> >>>>
>> >>>>> you can find the source code of the examples here:
>> >>>>> https://github.com/apache/flink/blob/master/flink-examples/
>> >>>>> flink-examples-streaming/src/main/java/org/apache/flink/
>> >>>>> streaming/examples/wordcount/WordCount.java
>> >>>>>
>> >>>>> A general introduction how the cluster execution works can be found
>> >>>>> here:
>> >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>> >>>>> concepts/programming-model.html#programs-and-dataflows
>> >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>> >>>>> concepts/runtime.html
>> >>>>>
>> >>>>> It might also be helpful to have a look at the web interface which
>> can
>> >>>>> show you a nice graph of the job.
>> >>>>>
>> >>>>> I hope this helps. Feel free to ask further questions.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Timo
>> >>>>>
>> >>>>>
>> >>>>> Am 07.08.17 um 14:00 schrieb P. Ramanjaneya Reddy:
>> >>>>>
>> >>>>> Hello Everyone,
>> >>>>>
>> >>>>> I have followed the steps specified below link to Install & Run
>> Apache
>> >>>>>> Flink on Multi-node Cluster.
>> >>>>>>
>> >>>>>> http://data-flair.training/blogs/install-run-deploy-flink-
>> >>>>>> multi-node-cluster/
>> >>>>>>     used flink-1.3.2-bin-hadoop27-scala_2.10.tgz for install
>> >>>>>>
>> >>>>>> using the command
>> >>>>>>     " bin/flink run
>> >>>>>> /home/root1/NAI/Tools/BEAM/Flink_Cluster/rama/flink/examples
>> >>>>>> /streaming/WordCount.jar"
>> >>>>>> able to run wordcount, but where can i see which input consider and
>> >>>>>> output
>> >>>>>> generated?
>> >>>>>>
>> >>>>>> and how can i specify the input and output paths?
>> >>>>>>
>> >>>>>> I'm trying to understand how the wordcount will work using
>> Multi-node
>> >>>>>> Cluster.?
>> >>>>>>
>> >>>>>> any suggestions will help me further understanding?
>> >>>>>>
>> >>>>>> Thanks & Regards,
>> >>>>>> Ramanji.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >
>>
>
>

Reply via email to