Re: Flink execution time benchmark.
Hi Saleh,
The new web interface in Flink 0.10 has also a REST API that you can use
for querying job information.
GET http://localhost:8081/jobs/83547b683ad5b388355a49911168fbc7
will give you the following JSON object:
{
"jid":"83547b683ad5b388355a49911168fbc7",
"name":"com.dataartisans.Forever",
"state":"RUNNING",
"start-time":1447947758093,
"end-time":-1,
"duration":89016,
"now":1447947847109,
"timestamps":{
"CREATED":1447947758093,
"RUNNING":1447947758106,
"FAILING":0,
"FAILED":0,
"CANCELLING":0,
"CANCELED":0,
"FINISHED":0,
"RESTARTING":0
},
"vertices":[
{
"id":"6005df847512e1ce9d49f591423a60f0",
"name":"Source: Custom Source -> Flat Map -> Sink: Unnamed",
"parallelism":1,
"status":"RUNNING",
"start-time":1447947758109,
"end-time":-1,
"duration":89000,
"tasks":{
"CREATED":0,
"SCHEDULED":0,
"DEPLOYING":0,
"RUNNING":1,
"FINISHED":0,
"CANCELING":0,
"CANCELED":0,
"FAILED":0
},
"metrics":{
"read-bytes":0,
"write-bytes":0,
"read-records":0,
"write-records":0
}
}
],
"status-counts":{
"CREATED":0,
"SCHEDULED":0,
"DEPLOYING":0,
"RUNNING":1,
"FINISHED":0,
"CANCELING":0,
"CANCELED":0,
"FAILED":0
},
"plan":{
"jid":"83547b683ad5b388355a49911168fbc7",
"name":"com.dataartisans.Forever",
"nodes":[
{
"id":"6005df847512e1ce9d49f591423a60f0",
"parallelism":1,
"operator":"(not set)",
"operator_strategy":"(not set)",
"description":"Source: Custom Source -> Flat Map -> Sink:
Unnamed",
"optimizer_properties":{
}
}
]
}
}
With that, you can compute the difference between "start-time" and "now".
Regards,
Robert
On Wed, Nov 18, 2015 at 10:17 PM, Saleh wrote:
> Hi rmetzger0,
>
> Thanx for the response. I didn't know that I had to register before I could
> receive responses for my posts.
> Now I am registered. But the problem is not resolved yet. I know it might
> not be intuitive to get execution time from a long running streaming job
> but
> it is possible to get total execution time after let say I stop the program
> from running. Can I programatically compute this information? Or can I
> retrieve it from Flink web UI?
>
> cheers.
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-execution-time-benchmark-tp3258p3573.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: Flink execution time benchmark.
Hi rmetzger0, Thanx for the response. I didn't know that I had to register before I could receive responses for my posts. Now I am registered. But the problem is not resolved yet. I know it might not be intuitive to get execution time from a long running streaming job but it is possible to get total execution time after let say I stop the program from running. Can I programatically compute this information? Or can I retrieve it from Flink web UI? cheers. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-execution-time-benchmark-tp3258p3573.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink execution time benchmark.
Hi Saleh, I'm sorry that nobody replied to your message. I think you were not subscribed to the user@flink.apache.org. mailing list when you posted this question. Therefore, we Flink community didn't receive your message. Did you resolve the issue in the meantime or are you still seeking for help? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-execution-time-benchmark-tp3258p3397.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink execution time benchmark
Hi Giacomo,
You are currently using the Flink Streaming API. Is that your intention or
would you like to measure batch execution?
Regarding your code: StreamExecutionEnvironment.readTextStream(filePath)
monitors a file/directory and streams the updates to that location [1] -
potentially indefinitely, so that job is not expected to stop.
If you wanted to read a text file with the streaming API you would need to
use the StreamExecutionEnvironment.readTextFile(filePath) function, the
same for the batch API is ExecutionEnvironment.readTextFile(filePath).
In case you wanted to measure the performance of the streaming framework I
personally would not necessarily recommend reading from a file as you
benchmark will be potentially disk I/O bounded. Reading from a local socket
or even generating data in a SourceFunction might be more beneficial. For
the batch version the file is of course a standard input.
After a pull request that I have just merged both the batch and the
streaming APIs env.execute methods return a JobExecutionResult from which
you can get the execution time as Robert suggested. [1] To get that please
depend on the latest master, 0.9-SNAPSHOT. In case you are testing with an
older stable version you can measure time from the command line:
time (bin/flink run your-jar.jar ... )
[1] https://github.com/apache/flink/pull/516
Best,
Marton
On Sun, Mar 22, 2015 at 5:56 PM, Giacomo Licari
wrote:
> Hi Robert,
> I can see the timestamp when the process starts but it seems doesn't stop
> executing.
>
> I have:
> DataStream dataStream = env
> .readTextStream(filePath)
> .flatMap(new Splitter());
>
> WindowedDataStream dataWindow =
> dataStream.window(Time.of(5, TimeUnit.SECONDS)).every(Time.of(5,
> TimeUnit.SECONDS));
>
> DataStream res = dataWindow.reduceGroup(new Reducer());
> env.execute("Flink Processor");
>
> Thank you,
> Giacomo
>
> On Sat, Mar 21, 2015 at 1:19 PM, Robert Metzger
> wrote:
>
>> Hi,
>>
>> The execute() method returns an execution result object that also
>> contains the runtime of the job.
>>
>>
>>
>> Sent from my iPhone
>>
>> > On 21.03.2015, at 11:09, Giacomo Licari
>> wrote:
>> >
>> > Hi guys,
>> > I'm trying to execute a benchmark reading a 1GB file and applying a
>> reduce function.
>> >
>> > I'd like to get the overall execution time.
>> >
>> > I put that code before applying the reduce function:
>> > long startTime = System.currentTimeMillis();
>> >
>> > and after env.execute("Flink Processor"); I put the code to print out
>> the overall time, but it doen't work.
>> >
>> > Did you try something about?
>> >
>> > Thanks a lot,
>> > Giacomo
>>
>
>
Re: Flink execution time benchmark
Hi Robert,
I can see the timestamp when the process starts but it seems doesn't stop
executing.
I have:
DataStream dataStream = env
.readTextStream(filePath)
.flatMap(new Splitter());
WindowedDataStream dataWindow =
dataStream.window(Time.of(5, TimeUnit.SECONDS)).every(Time.of(5,
TimeUnit.SECONDS));
DataStream res = dataWindow.reduceGroup(new Reducer());
env.execute("Flink Processor");
Thank you,
Giacomo
On Sat, Mar 21, 2015 at 1:19 PM, Robert Metzger wrote:
> Hi,
>
> The execute() method returns an execution result object that also contains
> the runtime of the job.
>
>
>
> Sent from my iPhone
>
> > On 21.03.2015, at 11:09, Giacomo Licari
> wrote:
> >
> > Hi guys,
> > I'm trying to execute a benchmark reading a 1GB file and applying a
> reduce function.
> >
> > I'd like to get the overall execution time.
> >
> > I put that code before applying the reduce function:
> > long startTime = System.currentTimeMillis();
> >
> > and after env.execute("Flink Processor"); I put the code to print out
> the overall time, but it doen't work.
> >
> > Did you try something about?
> >
> > Thanks a lot,
> > Giacomo
>
Re: Flink execution time benchmark
Hi,
The execute() method returns an execution result object that also contains the
runtime of the job.
Sent from my iPhone
> On 21.03.2015, at 11:09, Giacomo Licari wrote:
>
> Hi guys,
> I'm trying to execute a benchmark reading a 1GB file and applying a reduce
> function.
>
> I'd like to get the overall execution time.
>
> I put that code before applying the reduce function:
> long startTime = System.currentTimeMillis();
>
> and after env.execute("Flink Processor"); I put the code to print out the
> overall time, but it doen't work.
>
> Did you try something about?
>
> Thanks a lot,
> Giacomo
Flink execution time benchmark
Hi guys,
I'm trying to execute a benchmark reading a 1GB file and applying a reduce
function.
I'd like to get the overall execution time.
I put that code before applying the reduce function:
long startTime = System.currentTimeMillis();
and after env.execute("Flink Processor"); I put the code to print out the
overall time, but it doen't work.
Did you try something about?
Thanks a lot,
Giacomo
