Hi Timo,

How to make access the files across TM?

Thanks & Regards,
Ramanji.

On Mon, Aug 7, 2017 at 7:45 PM, Timo Walther <twal...@apache.org> wrote:

> Flink is a distributed software for clusters. You need something like a
> distributed file system. So that input file and output files can be
> accessed from all nodes.
>
> Each TM has a log directory where the execution logs are stored.
>
> You can set additional properties to your output format by importing the
> code in your IDE.
>
> Am 07.08.17 um 16:03 schrieb P. Ramanjaneya Reddy:
>
> Hi Timo,
>> Problem is resolved after copy input file to all tasks managers.
>>
>> and where should generate outputfile? Is it in jobmanager or task manager?
>>
>> Where can i see the execution logs to understand how word count done each
>> task manager?
>>
>>
>> By the way any option to overwride...?
>>
>> 08/07/2017 19:27:00 Keyed Aggregation -> Sink: Unnamed(1/1) switched to
>> FAILED
>> java.io.IOException: File or directory already exists. Existing files and
>> directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode
>> to
>> overwrite existing files and directories.
>> at
>> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileS
>> ystem.java:763)
>> at
>> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.initOutP
>> athLocalFS(SafetyNetWrapperFileSystem.java:135)
>> at
>> org.apache.flink.api.common.io.FileOutputFormat.open(FileOut
>> putFormat.java:231)
>> at
>> org.apache.flink.api.java.io.TextOutputFormat.open(TextOutpu
>> tFormat.java:78)
>> at
>> org.apache.flink.streaming.api.functions.sink.OutputFormatSi
>> nkFunction.open(OutputFormatSinkFunction.java:61)
>> at
>> org.apache.flink.api.common.functions.util.FunctionUtils.ope
>> nFunction(FunctionUtils.java:36)
>> at
>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOp
>> erator.open(AbstractUdfStreamOperator.java:111)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO
>> perators(StreamTask.java:376)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>> StreamTask.java:253)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> On Mon, Aug 7, 2017 at 6:49 PM, Timo Walther <twal...@apache.org> wrote:
>>
>> Make sure that the file exists and is accessible from all Flink tasks
>>> managers.
>>>
>>>
>>> Am 07.08.17 um 14:35 schrieb P. Ramanjaneya Reddy:
>>>
>>> Thank you Timo.
>>>>
>>>>
>>>> root1@root1-HP-EliteBook-840-G2:~/NAI/Tools/BEAM/Flink_Clust
>>>> er/rama/flink$
>>>> *./bin/flink
>>>> run ./examples/streaming/WordCount.jar --input
>>>> file:///home/root1/hamlet.txt --output file:///home/root1/wordcount_o
>>>> ut*
>>>>
>>>>
>>>>
>>>> Execution of worcountjar gives error...
>>>>
>>>> 08/07/2017 18:03:16 Source: Custom File Source(1/1) switched to FAILED
>>>> java.io.FileNotFoundException: The provided file path
>>>> file:/home/root1/hamlet.txt does not exist.
>>>> at
>>>> org.apache.flink.streaming.api.functions.source.ContinuousFi
>>>> leMonitoringFunction.run(ContinuousFileMonitoringFunction.java:192)
>>>> at
>>>> org.apache.flink.streaming.api.operators.StreamSource.run(
>>>> StreamSource.java:87)
>>>> at
>>>> org.apache.flink.streaming.api.operators.StreamSource.run(
>>>> StreamSource.java:55)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.
>>>> run(SourceStreamTask.java:95)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>>>> StreamTask.java:263)
>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>>
>>>> On Mon, Aug 7, 2017 at 5:56 PM, Timo Walther <twal...@apache.org>
>>>> wrote:
>>>>
>>>> Hi Ramanji,
>>>>
>>>>> you can find the source code of the examples here:
>>>>> https://github.com/apache/flink/blob/master/flink-examples/
>>>>> flink-examples-streaming/src/main/java/org/apache/flink/
>>>>> streaming/examples/wordcount/WordCount.java
>>>>>
>>>>> A general introduction how the cluster execution works can be found
>>>>> here:
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>>>>> concepts/programming-model.html#programs-and-dataflows
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>>>>> concepts/runtime.html
>>>>>
>>>>> It might also be helpful to have a look at the web interface which can
>>>>> show you a nice graph of the job.
>>>>>
>>>>> I hope this helps. Feel free to ask further questions.
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 07.08.17 um 14:00 schrieb P. Ramanjaneya Reddy:
>>>>>
>>>>> Hello Everyone,
>>>>>
>>>>> I have followed the steps specified below link to Install & Run Apache
>>>>>> Flink on Multi-node Cluster.
>>>>>>
>>>>>> http://data-flair.training/blogs/install-run-deploy-flink-
>>>>>> multi-node-cluster/
>>>>>>     used flink-1.3.2-bin-hadoop27-scala_2.10.tgz for install
>>>>>>
>>>>>> using the command
>>>>>>     " bin/flink run
>>>>>> /home/root1/NAI/Tools/BEAM/Flink_Cluster/rama/flink/examples
>>>>>> /streaming/WordCount.jar"
>>>>>> able to run wordcount, but where can i see which input consider and
>>>>>> output
>>>>>> generated?
>>>>>>
>>>>>> and how can i specify the input and output paths?
>>>>>>
>>>>>> I'm trying to understand how the wordcount will work using Multi-node
>>>>>> Cluster.?
>>>>>>
>>>>>> any suggestions will help me further understanding?
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> Ramanji.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Reply via email to