Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Khachatryan Roman
Thanks a lot for your replies! Yes, feedback is very much appreciated! Especially regarding the approach in general. I think that's a good idea to discuss some details in the follow-up docs or tickets (but I'd be happy to discuss it here as well). As for the PoC, I hope we'll publish it soon

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Yuan Mei
Big +1 onto this FLIP! Great to see it is stepping forward since this idea is discussed for quite a while. :-) 1. I totally agree that the critical part is the overhead added during normal state updates (forward additional state updates to log as well as state updates itself). Once we have this

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Stephan Ewen
+1 to this FLIP in general. I like the general idea very much (full disclosure, have been involved in the discussions and drafting of the design for a while, so I am not a new/neutral reviewer here). One thing I would like to see us do here, is reaching out to users early with this, and

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Piotr Nowojski
Hi Roman, +1 from my side on this proposal. Also big +1 for the recent changes in this FLIP in the motivation and high level overview sections. For me there are quite a bit of unanswered things around how to actually implement the proposed changes and especially how to integrate it with the

[DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-14 Thread Khachatryan Roman
Hi devs, I'd like to start a discussion of FLIP-158: Generalized incremental checkpoints [1] FLIP motivation: Low end-to-end latency is a much-demanded property in many Flink setups. With exactly-once, this latency depends on checkpoint interval/duration which in turn is defined by the slowest