Re: Measure Latency from source to sink

[Hequn Cheng]

Hi antonio, latency is exposed via a metric. You can find each operator's
latency through flink UI(Overview->Task Metrics -> select the task, for
example select the sink -> Add metric -> find latency metric)

On Tue, Jun 26, 2018 at 11:18 PM, antonio saldivar 
wrote:

> Hello thank you
>
> I also was trying using Flink UI Metrics on version 1.4.2  
> *env.getConfig().setLatencyTrackingInterval(1000L),
> *But looks like is not displaying anything
>
> El mar., 26 jun. 2018 a las 10:45, zhangminglei (<18717838...@163.com>)
> escribió:
>
>> Hi, You can do that but it does not makes sense in general. But you can
>> do that by flink, storm, spark streaming or structured streaming.  And make
>> a compare the latency under different framework.
>>
>> Cheers
>> Minglei
>>
>> 在 2018年6月26日，下午9:36，antonio saldivar  写道：
>>
>> Hello Thank you for the feedback,
>>
>> Well for now I just Want to measure the time that takes form Source to
>> Sink each transaction add the start and end time in mills
>>
>>
>>
>> El mar., 26 jun. 2018 a las 5:19, zhangminglei (<18717838...@163.com>)
>> escribió:
>>
>>> Hi，Antonio
>>>
>>> Usually, the measurement of delay is for specific business I think it is
>>> more reasonable. What I understand of latency from my experience is data
>>> preparation time plus query calculation time. It is like an end to end
>>> latency test. Hopes this can help you. Not point to the latency of flink
>>>
>>> Cheers
>>> Minglei
>>>
>>>
>>> > 在 2018年6月26日，上午5:23，antonio saldivar  写道：
>>> >
>>> > Hello
>>> >
>>> > I am trying to measure the latency of each transaction traveling
>>> across the system as a DataSource I have a Kafka consumer and I would like
>>> to measure the time that takes from the Source to Sink. Does any one has an
>>> example?.
>>> >
>>> > Thank you
>>> > Best Regards
>>>
>>>
>>>
>>

Re: Measure Latency from source to sink

[antonio saldivar]

Hello thank you

I also was trying using Flink UI Metrics on version 1.4.2
*env.getConfig().setLatencyTrackingInterval(1000L),
*But looks like is not displaying anything

El mar., 26 jun. 2018 a las 10:45, zhangminglei (<18717838...@163.com>)
escribió:

> Hi, You can do that but it does not makes sense in general. But you can do
> that by flink, storm, spark streaming or structured streaming.  And make a
> compare the latency under different framework.
>
> Cheers
> Minglei
>
> 在 2018年6月26日，下午9:36，antonio saldivar  写道：
>
> Hello Thank you for the feedback,
>
> Well for now I just Want to measure the time that takes form Source to
> Sink each transaction add the start and end time in mills
>
>
>
> El mar., 26 jun. 2018 a las 5:19, zhangminglei (<18717838...@163.com>)
> escribió:
>
>> Hi，Antonio
>>
>> Usually, the measurement of delay is for specific business I think it is
>> more reasonable. What I understand of latency from my experience is data
>> preparation time plus query calculation time. It is like an end to end
>> latency test. Hopes this can help you. Not point to the latency of flink
>>
>> Cheers
>> Minglei
>>
>>
>> > 在 2018年6月26日，上午5:23，antonio saldivar  写道：
>> >
>> > Hello
>> >
>> > I am trying to measure the latency of each transaction traveling across
>> the system as a DataSource I have a Kafka consumer and I would like to
>> measure the time that takes from the Source to Sink. Does any one has an
>> example?.
>> >
>> > Thank you
>> > Best Regards
>>
>>
>>
>

Re: Measure Latency from source to sink

[zhangminglei]

Hi, You can do that but it does not makes sense in general. But you can do that 
by flink, storm, spark streaming or structured streaming.  And make a compare 
the latency under different framework.

Cheers
Minglei

> 在 2018年6月26日，下午9:36，antonio saldivar  写道：
> 
> Hello Thank you for the feedback,
> 
> Well for now I just Want to measure the time that takes form Source to Sink 
> each transaction add the start and end time in mills
> 
> 
> 
> El mar., 26 jun. 2018 a las 5:19, zhangminglei (<18717838...@163.com 
> >) escribió:
> Hi，Antonio
> 
> Usually, the measurement of delay is for specific business I think it is more 
> reasonable. What I understand of latency from my experience is data 
> preparation time plus query calculation time. It is like an end to end 
> latency test. Hopes this can help you. Not point to the latency of flink
> 
> Cheers
> Minglei
> 
> 
> > 在 2018年6月26日，上午5:23，antonio saldivar  > > 写道：
> > 
> > Hello
> > 
> > I am trying to measure the latency of each transaction traveling across the 
> > system as a DataSource I have a Kafka consumer and I would like to measure 
> > the time that takes from the Source to Sink. Does any one has an example?.
> > 
> > Thank you
> > Best Regards
> 
> 

Re: Measure Latency from source to sink

[antonio saldivar]

Hello Thank you for the feedback,

Well for now I just Want to measure the time that takes form Source to Sink
each transaction add the start and end time in mills



El mar., 26 jun. 2018 a las 5:19, zhangminglei (<18717838...@163.com>)
escribió:

> Hi，Antonio
>
> Usually, the measurement of delay is for specific business I think it is
> more reasonable. What I understand of latency from my experience is data
> preparation time plus query calculation time. It is like an end to end
> latency test. Hopes this can help you. Not point to the latency of flink
>
> Cheers
> Minglei
>
>
> > 在 2018年6月26日，上午5:23，antonio saldivar  写道：
> >
> > Hello
> >
> > I am trying to measure the latency of each transaction traveling across
> the system as a DataSource I have a Kafka consumer and I would like to
> measure the time that takes from the Source to Sink. Does any one has an
> example?.
> >
> > Thank you
> > Best Regards
>
>
>

Re: Measure Latency from source to sink

[zhangminglei]

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more 
reasonable. What I understand of latency from my experience is data preparation 
time plus query calculation time. It is like an end to end latency test. Hopes 
this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar  写道：
> 
> Hello
> 
> I am trying to measure the latency of each transaction traveling across the 
> system as a DataSource I have a Kafka consumer and I would like to measure 
> the time that takes from the Source to Sink. Does any one has an example?.
> 
> Thank you
> Best Regards

Re: Measure Latency from source to sink

[Fabian Hueske]

Hi,

Measuring latency is tricky and you have to be careful about what you
measure.
Aggregations like window operators make things even more difficult because
you need to decide which timestamp(s) to forward (smallest?, largest?, all?)
Depending on the operation, the measurement code might even add to the
overall latency.
Also, the clock of the nodes in your cluster might not be totally in sync.

Best, Fabian

2018-06-26 4:00 GMT+02:00 antonio saldivar :

> Thank you very much
>
> I already did #2 but ate the moment i print te output as i am using a
> trigger alert and evaluete the window it replace me the toString values to
> null or 0 and only prints the ones saved in my accumulator and the keyBy
> value
>
> On Mon, Jun 25, 2018, 9:22 PM Hequn Cheng  wrote:
>
>> Hi antonio,
>>
>> I see two options to solve your problem.
>> 1. Enable the latency tracking[1]. But you have to pay attention to it's
>> mechanism, for example, a) the sources only *periodically* emit a
>> special record and b) the latency markers are not accounting for the time
>> user records spend in operators as they are bypassing them.
>> 2. Add a time field to each of your record. Each time a record comes in
>> from the source, write down the time(t1), so that we can get the latency at
>> sink(t2) by  t2 - t1.
>>
>> Hope this helps.
>> Hequn
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-
>> master/monitoring/metrics.html#latency-tracking
>>
>> On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar 
>> wrote:
>>
>>> Hello
>>>
>>> I am trying to measure the latency of each transaction traveling across
>>> the system as a DataSource I have a Kafka consumer and I would like to
>>> measure the time that takes from the Source to Sink. Does any one has an
>>> example?.
>>>
>>> Thank you
>>> Best Regards
>>>
>>
>>

Re: Measure Latency from source to sink

[antonio saldivar]

Thank you very much

I already did #2 but ate the moment i print te output as i am using a
trigger alert and evaluete the window it replace me the toString values to
null or 0 and only prints the ones saved in my accumulator and the keyBy
value

On Mon, Jun 25, 2018, 9:22 PM Hequn Cheng  wrote:

> Hi antonio,
>
> I see two options to solve your problem.
> 1. Enable the latency tracking[1]. But you have to pay attention to it's
> mechanism, for example, a) the sources only *periodically* emit a special
> record and b) the latency markers are not accounting for the time user
> records spend in operators as they are bypassing them.
> 2. Add a time field to each of your record. Each time a record comes in
> from the source, write down the time(t1), so that we can get the latency at
> sink(t2) by  t2 - t1.
>
> Hope this helps.
> Hequn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#latency-tracking
>
> On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar 
> wrote:
>
>> Hello
>>
>> I am trying to measure the latency of each transaction traveling across
>> the system as a DataSource I have a Kafka consumer and I would like to
>> measure the time that takes from the Source to Sink. Does any one has an
>> example?.
>>
>> Thank you
>> Best Regards
>>
>
>

Re: Measure Latency from source to sink

[Hequn Cheng]

Hi antonio,

I see two options to solve your problem.
1. Enable the latency tracking[1]. But you have to pay attention to it's
mechanism, for example, a) the sources only *periodically* emit a special
record and b) the latency markers are not accounting for the time user
records spend in operators as they are bypassing them.
2. Add a time field to each of your record. Each time a record comes in
from the source, write down the time(t1), so that we can get the latency at
sink(t2) by  t2 - t1.

Hope this helps.
Hequn

[1]
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#latency-tracking

On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar 
wrote:

> Hello
>
> I am trying to measure the latency of each transaction traveling across
> the system as a DataSource I have a Kafka consumer and I would like to
> measure the time that takes from the Source to Sink. Does any one has an
> example?.
>
> Thank you
> Best Regards
>