Thanks. I didn't know we could set that.
On Fri, May 13, 2016 at 12:44 PM, Stephan Ewen wrote:
> You can use the checkpoint mode to "at least once".
> That way, barriers never block.
>
> On Fri, May 13, 2016 at 6:05 PM, Srikanth wrote:
>
>> I have a follow up. Is there a recommendation of list
You can use the checkpoint mode to "at least once".
That way, barriers never block.
On Fri, May 13, 2016 at 6:05 PM, Srikanth wrote:
> I have a follow up. Is there a recommendation of list of knobs that can be
> tuned if at least once guarantee while handling failure is good enough?
> For cases
I have a follow up. Is there a recommendation of list of knobs that can be
tuned if at least once guarantee while handling failure is good enough?
For cases like alert generation, non idempotent sink, etc where the system
can live with duplicates or has other mechanism to handle them.
Srikanth
On
Thanks Matthias & Stephan!
Yes, if we choose to fail checkpoint on expiry, we can restore from
previous checkpoint.
Looking forward to read the new design proposal.
Srikanth
On Fri, May 13, 2016 at 8:09 AM, Stephan Ewen wrote:
> Hi Srikanth!
>
> That is an interesting idea.
> I have it on my
Hi Srikanth!
That is an interesting idea.
I have it on my mind to create a design doc for checkpointing improvements.
That could be added as a proposal there.
I hope I'll be able to start with that design doc next week.
Greetings,
Stephan
On Fri, May 13, 2016 at 1:35 PM, Matthias J. Sax wrote
I don't think barries can "expire" as of now. Might be a nice idea
thought -- I don't know if this might be a problem in production.
Furthermore, I want to point out, that an "expiring checkpoint" would
not break exactly-once processing, as the latest successful checkpoint
can always be used to re
Hello,
I was reading about Flink's checkpoint and wanted to check if I correctly
understood the usage of barriers for exactly once processing.
1) Operator does alignment by buffering records coming after a barrier
until it receives barrier from all upstream operators instances.
2) Barrier is alw
