Re: Checkpointing Event Time Watermarks

[vijay kansal]

Hi Xingcan

We are receiving events from a no. of independent data sources and hence,
data arriving into our Flink topology (via Kafka) would be out of order.

We are creating 1-min event time windows in our Flink topology and
generating event time watermarks as (current event time - some threshold
(30 seconds)) at the source operator.

In case a few events arrive after the set threshold, those events are
simply ignored (which is ok in our case, because most of the events
belonging to that minute would have already arrived and got processed in
the corresponding window).

Now, the problem is that in case the program crashes (for whatever reason)
and is then resumed again from the last successful checkpoint, out of order
arriving events would trigger execution of past (already processed) windows
(with only a minuscule of events in that window) overriding results of
prev. computation of that window.

In case Flink had checkpointed event time watermarks, this problem would
not have occurred.

So, I am wondering if there is a way to enforce event time watermarks'
checkpointing in Flink...











Vijay Kansal
Software Development Engineer
LimeRoad

On Tue, Feb 27, 2018 at 6:54 PM, Xingcan Cui  wrote:

> Hi Vijay,
>
> normally, maybe there’s no need to checkpoint the event times / watermarks
> since they are automatically generated based on the records. What’s your
> intention?
>
> Best,
> Xingcan
>
> > On 27 Feb 2018, at 8:50 PM, vijay kansal  wrote:
> >
> > Hi All
> >
> > Is there a way to checkpoint event time watermarks in Flink ?
> >
> > I tries searching for this, but could not figure out...
> >
> >
> > Vijay Kansal
> > Software Development Engineer
> > LimeRoad
>
>

Re: Checkpointing Event Time Watermarks

[Xingcan Cui]

Hi Vijay,

normally, maybe there’s no need to checkpoint the event times / watermarks 
since they are automatically generated based on the records. What’s your 
intention?

Best,
Xingcan

> On 27 Feb 2018, at 8:50 PM, vijay kansal  wrote:
> 
> Hi All
> 
> Is there a way to checkpoint event time watermarks in Flink ?
> 
> I tries searching for this, but could not figure out...
> 
> 
> Vijay Kansal
> Software Development Engineer
> LimeRoad

Checkpointing Event Time Watermarks

[vijay kansal]

Hi All

Is there a way to checkpoint event time watermarks in Flink ?

I tries searching for this, but could not figure out...


Vijay Kansal
Software Development Engineer
LimeRoad