Re: Barriers at work

[Stephan Ewen]

You can use the checkpoint mode to "at least once".
That way, barriers never block.

On Fri, May 13, 2016 at 6:05 PM, Srikanth  wrote:

> I have a follow up. Is there a recommendation of list of knobs that can be
> tuned if at least once guarantee while handling failure is good enough?
> For cases like alert generation, non idempotent sink, etc where the system
> can live with duplicates or has other mechanism to handle them.
>
> Srikanth
>
> On Fri, May 13, 2016 at 8:09 AM, Stephan Ewen  wrote:
>
>> Hi Srikanth!
>>
>> That is an interesting idea.
>> I have it on my mind to create a design doc for checkpointing
>> improvements. That could be added as a proposal there.
>>
>> I hope I'll be able to start with that design doc next week.
>>
>> Greetings,
>> Stephan
>>
>>
>> On Fri, May 13, 2016 at 1:35 PM, Matthias J. Sax 
>> wrote:
>>
>>> I don't think barries can "expire" as of now. Might be a nice idea
>>> thought -- I don't know if this might be a problem in production.
>>>
>>> Furthermore, I want to point out, that an "expiring checkpoint" would
>>> not break exactly-once processing, as the latest successful checkpoint
>>> can always be used to recover correctly. Only the recovery-time would be
>>> increase. because if a "barrier expires" and no checkpoint can be
>>> stored, more data has to be replayed using the "old" checkpoint".
>>>
>>>
>>> -Matthias
>>>
>>> On 05/12/2016 09:21 PM, Srikanth wrote:
>>> > Hello,
>>> >
>>> > I was reading about Flink's checkpoint and wanted to check if I
>>> > correctly understood the usage of barriers for exactly once processing.
>>> >  1) Operator does alignment by buffering records coming after a barrier
>>> > until it receives barrier from all upstream operators instances.
>>> >  2) Barrier is always preceded by a watermark to trigger processing all
>>> > windows that are complete.
>>> >  3) Records in windows that are not triggered are also saved as part of
>>> > checkpoint. These windows are repopulated when restoring from
>>> checkpoints.
>>> >
>>> > In production setups, were there any cases where alignment during
>>> > checkpointing caused unacceptable latency?
>>> > If so, is there a way to indicate say wait for a MAX 100 ms? That way
>>> we
>>> > have exactly-once in most situations but prefer at least once over
>>> > higher latency in corner cases.
>>> >
>>> > Srikanth
>>>
>>>
>>
>

Re: Barriers at work

[Srikanth]

I have a follow up. Is there a recommendation of list of knobs that can be
tuned if at least once guarantee while handling failure is good enough?
For cases like alert generation, non idempotent sink, etc where the system
can live with duplicates or has other mechanism to handle them.

Srikanth

On Fri, May 13, 2016 at 8:09 AM, Stephan Ewen  wrote:

> Hi Srikanth!
>
> That is an interesting idea.
> I have it on my mind to create a design doc for checkpointing
> improvements. That could be added as a proposal there.
>
> I hope I'll be able to start with that design doc next week.
>
> Greetings,
> Stephan
>
>
> On Fri, May 13, 2016 at 1:35 PM, Matthias J. Sax  wrote:
>
>> I don't think barries can "expire" as of now. Might be a nice idea
>> thought -- I don't know if this might be a problem in production.
>>
>> Furthermore, I want to point out, that an "expiring checkpoint" would
>> not break exactly-once processing, as the latest successful checkpoint
>> can always be used to recover correctly. Only the recovery-time would be
>> increase. because if a "barrier expires" and no checkpoint can be
>> stored, more data has to be replayed using the "old" checkpoint".
>>
>>
>> -Matthias
>>
>> On 05/12/2016 09:21 PM, Srikanth wrote:
>> > Hello,
>> >
>> > I was reading about Flink's checkpoint and wanted to check if I
>> > correctly understood the usage of barriers for exactly once processing.
>> >  1) Operator does alignment by buffering records coming after a barrier
>> > until it receives barrier from all upstream operators instances.
>> >  2) Barrier is always preceded by a watermark to trigger processing all
>> > windows that are complete.
>> >  3) Records in windows that are not triggered are also saved as part of
>> > checkpoint. These windows are repopulated when restoring from
>> checkpoints.
>> >
>> > In production setups, were there any cases where alignment during
>> > checkpointing caused unacceptable latency?
>> > If so, is there a way to indicate say wait for a MAX 100 ms? That way we
>> > have exactly-once in most situations but prefer at least once over
>> > higher latency in corner cases.
>> >
>> > Srikanth
>>
>>
>

Re: Barriers at work

[Srikanth]

Thanks Matthias & Stephan!

Yes, if we choose to fail checkpoint on expiry, we can restore from
previous checkpoint.

Looking forward to read the new design proposal.

Srikanth


On Fri, May 13, 2016 at 8:09 AM, Stephan Ewen  wrote:

> Hi Srikanth!
>
> That is an interesting idea.
> I have it on my mind to create a design doc for checkpointing
> improvements. That could be added as a proposal there.
>
> I hope I'll be able to start with that design doc next week.
>
> Greetings,
> Stephan
>
>
> On Fri, May 13, 2016 at 1:35 PM, Matthias J. Sax  wrote:
>
>> I don't think barries can "expire" as of now. Might be a nice idea
>> thought -- I don't know if this might be a problem in production.
>>
>> Furthermore, I want to point out, that an "expiring checkpoint" would
>> not break exactly-once processing, as the latest successful checkpoint
>> can always be used to recover correctly. Only the recovery-time would be
>> increase. because if a "barrier expires" and no checkpoint can be
>> stored, more data has to be replayed using the "old" checkpoint".
>>
>>
>> -Matthias
>>
>> On 05/12/2016 09:21 PM, Srikanth wrote:
>> > Hello,
>> >
>> > I was reading about Flink's checkpoint and wanted to check if I
>> > correctly understood the usage of barriers for exactly once processing.
>> >  1) Operator does alignment by buffering records coming after a barrier
>> > until it receives barrier from all upstream operators instances.
>> >  2) Barrier is always preceded by a watermark to trigger processing all
>> > windows that are complete.
>> >  3) Records in windows that are not triggered are also saved as part of
>> > checkpoint. These windows are repopulated when restoring from
>> checkpoints.
>> >
>> > In production setups, were there any cases where alignment during
>> > checkpointing caused unacceptable latency?
>> > If so, is there a way to indicate say wait for a MAX 100 ms? That way we
>> > have exactly-once in most situations but prefer at least once over
>> > higher latency in corner cases.
>> >
>> > Srikanth
>>
>>
>

Re: Barriers at work

[Stephan Ewen]

Hi Srikanth!

That is an interesting idea.
I have it on my mind to create a design doc for checkpointing improvements.
That could be added as a proposal there.

I hope I'll be able to start with that design doc next week.

Greetings,
Stephan


On Fri, May 13, 2016 at 1:35 PM, Matthias J. Sax  wrote:

> I don't think barries can "expire" as of now. Might be a nice idea
> thought -- I don't know if this might be a problem in production.
>
> Furthermore, I want to point out, that an "expiring checkpoint" would
> not break exactly-once processing, as the latest successful checkpoint
> can always be used to recover correctly. Only the recovery-time would be
> increase. because if a "barrier expires" and no checkpoint can be
> stored, more data has to be replayed using the "old" checkpoint".
>
>
> -Matthias
>
> On 05/12/2016 09:21 PM, Srikanth wrote:
> > Hello,
> >
> > I was reading about Flink's checkpoint and wanted to check if I
> > correctly understood the usage of barriers for exactly once processing.
> >  1) Operator does alignment by buffering records coming after a barrier
> > until it receives barrier from all upstream operators instances.
> >  2) Barrier is always preceded by a watermark to trigger processing all
> > windows that are complete.
> >  3) Records in windows that are not triggered are also saved as part of
> > checkpoint. These windows are repopulated when restoring from
> checkpoints.
> >
> > In production setups, were there any cases where alignment during
> > checkpointing caused unacceptable latency?
> > If so, is there a way to indicate say wait for a MAX 100 ms? That way we
> > have exactly-once in most situations but prefer at least once over
> > higher latency in corner cases.
> >
> > Srikanth
>
>

Re: Barriers at work

[Matthias J. Sax]

I don't think barries can "expire" as of now. Might be a nice idea
thought -- I don't know if this might be a problem in production.

Furthermore, I want to point out, that an "expiring checkpoint" would
not break exactly-once processing, as the latest successful checkpoint
can always be used to recover correctly. Only the recovery-time would be
increase. because if a "barrier expires" and no checkpoint can be
stored, more data has to be replayed using the "old" checkpoint".


-Matthias

On 05/12/2016 09:21 PM, Srikanth wrote:
> Hello,
> 
> I was reading about Flink's checkpoint and wanted to check if I
> correctly understood the usage of barriers for exactly once processing.
>  1) Operator does alignment by buffering records coming after a barrier
> until it receives barrier from all upstream operators instances.
>  2) Barrier is always preceded by a watermark to trigger processing all
> windows that are complete.
>  3) Records in windows that are not triggered are also saved as part of
> checkpoint. These windows are repopulated when restoring from checkpoints. 
> 
> In production setups, were there any cases where alignment during
> checkpointing caused unacceptable latency?
> If so, is there a way to indicate say wait for a MAX 100 ms? That way we
> have exactly-once in most situations but prefer at least once over
> higher latency in corner cases.
> 
> Srikanth



signature.asc
Description: OpenPGP digital signature