Hi Reuven, As Luke mentioned, at least there are some limitations around tracking watermark with flink cycles. I'm going to use State + Timer without flink cycle to support self-checkpoint. For dynamic split, we can either explore flink cycle approach or limit depth approach.
On Tue, Oct 6, 2020 at 5:33 PM Reuven Lax <[email protected]> wrote: > Aren't there some limitations associated with flink cycles? I seem to > remember various features that could not be used. I'm assuming that > watermarks are not supported across cycles, but is there anything else? > > On Tue, Oct 6, 2020 at 7:12 AM Maximilian Michels <[email protected]> wrote: > >> Thanks for starting the conversation. The two approaches both look good >> to me. Probably we want to start with approach #1 for all Runners to be >> able to support delaying bundles. Flink supports cycles and thus >> approach #2 would also be applicable and could be used to implement >> dynamic splitting. >> >> -Max >> >> On 05.10.20 23:13, Luke Cwik wrote: >> > Thanks Boyuan, I left a few comments. >> > >> > On Mon, Oct 5, 2020 at 11:12 AM Boyuan Zhang <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi team, >> > >> > I'm looking at adding self-checkpoint support to portable Flink >> > runner(BEAM-10940 >> > <https://issues.apache.org/jira/browse/BEAM-10940>) for both batch >> > and streaming. I summarized the problem that we want to solve and >> > proposed 2 potential approaches in this doc >> > < >> https://docs.google.com/document/d/1372B7HYxtcUYjZOnOM7OBTfSJ4CyFg_gaPD_NUxWClo/edit?usp=sharing >> >. >> > >> > I want to collect feedback on which approach is preferred and >> > anything that I have not taken into consideration yet but I should. >> > Many thanks to all your help! >> > >> > Boyuan >> > >> >
