Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-16 Thread Arvid Heise
eckpoint > > > time when back-pressure occurs, and I totally agree with that the > feature > > > could be implemented in steps. > > > > > > > > > -------------- > > > From:Roman Khachatryan > > > Send Time:2020 Mar. 12 (Thu.) 01:33 > > > To:dev

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-13 Thread Yu Li
--- > > From:Roman Khachatryan > > Send Time:2020 Mar. 12 (Thu.) 01:33 > > To:dev ; Zhijiang > > Subject:Re: [VOTE] [FLIP-76] Unaligned checkpoints > > > > +1 (non-binding) > > > > Regarding Yu's suggestion about *R

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Yingjie Cao
+1 (non-binding) Checkpoint timeout in cases of backpressure is hard to tune. I and our users ever spent lots of time on that. It is great to have this feature. Arvid Heise 于2020年3月10日周二 下午9:33写道: > Hi all, > > I would like to start the vote for FLIP-76 [1], which is discussed and > reached a

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Arvid Heise
ted in steps. > > > -- > From:Roman Khachatryan > Send Time:2020 Mar. 12 (Thu.) 01:33 > To:dev ; Zhijiang > Subject:Re: [VOTE] [FLIP-76] Unaligned checkpoints > > +1 (non-binding) > > Regarding Yu's suggestion about *Road

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Yun Gao
:2020 Mar. 12 (Thu.) 01:33 To:dev ; Zhijiang Subject:Re: [VOTE] [FLIP-76] Unaligned checkpoints +1 (non-binding) Regarding Yu's suggestion about *Roadmap* or *Future Work* section, I think it's a good idea. Currently, some MVP limitations are mentioned at the end of the document, so we can

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Roman Khachatryan
> > [1] > https://docs.google.com/document/d/16_MOQymzxrKvUHXh6QFr2AAXIKt_2vPUf8vzKy4H_tU/edit > > Best, > Zhijiang > > > -- > From:Piotr Nowojski > Send Time:2020 Mar. 11 (Wed.) 21:19 > To:dev > Sub

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Zhijiang
-- From:Piotr Nowojski Send Time:2020 Mar. 11 (Wed.) 21:19 To:dev Subject:Re: [VOTE] [FLIP-76] Unaligned checkpoints +1 (binding). Piotrek > On 11 Mar 2020, at 09:19, David Anderson wrote: > > +1 I like where this

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Piotr Nowojski
+1 (binding). Piotrek > On 11 Mar 2020, at 09:19, David Anderson wrote: > > +1 I like where this is headed. > > One question: during restore, it could happen that a new task manager is > configured with fewer or smaller buffers than was previously the case. How > will this be handled? > >

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread David Anderson
+1 I like where this is headed. One question: during restore, it could happen that a new task manager is configured with fewer or smaller buffers than was previously the case. How will this be handled? David On Wed, Mar 11, 2020 at 8:31 AM Arvid Heise wrote: > Hi Thomas, > > it's like you

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Arvid Heise
Hi Thomas, it's like you said. The first version will not support rescaling and mostly addresses the concerns about making little to no progress because of frequent crashes. The main reason is that we cannot guarantee the ordering of non-keyed data (and even keyed data in some weird cases) when

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-10 Thread Yu Li
+1 on the overall design and thanks for the efforts! I totally agree with the plan of implementing the MVP first. However, since the FLIP is for the whole feature instead of only MVP, how about adding a *Roadmap* or *Future Work* section to write down plans include (but not limited to): * Dynamic

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-10 Thread Thomas Weise
+1 Thanks for putting this together, looking forward to the experimental support in the next release. One clarification: since the MVP won't support rescaling, does it imply that savepoints will always use aligned checkpointing? If so, this would still block the user from taking a savepoint and

[VOTE] [FLIP-76] Unaligned checkpoints

2020-03-10 Thread Arvid Heise
Hi all, I would like to start the vote for FLIP-76 [1], which is discussed and reached a consensus in the discussion thread [2]. The vote will be open until March. 13th (72h), unless there is an objection or not enough votes. Thanks, Arvid [1]