This is what I was thinking of "Flink currently only provides processing guarantees for jobs without iterations. Enabling checkpointing on an iterative job causes an exception. In order to force checkpointing on an iterative program the user needs to set a special flag when enabling checkpointing: env.enableCheckpointing(interval, CheckpointingMode.EXACTLY_ONCE, force = true).
Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure." On Tue, Oct 6, 2020 at 5:44 PM Boyuan Zhang <[email protected]> wrote: > Hi Reuven, > > As Luke mentioned, at least there are some limitations around tracking > watermark with flink cycles. I'm going to use State + Timer without flink > cycle to support self-checkpoint. For dynamic split, we can either explore > flink cycle approach or limit depth approach. > > On Tue, Oct 6, 2020 at 5:33 PM Reuven Lax <[email protected]> wrote: > >> Aren't there some limitations associated with flink cycles? I seem to >> remember various features that could not be used. I'm assuming that >> watermarks are not supported across cycles, but is there anything else? >> >> On Tue, Oct 6, 2020 at 7:12 AM Maximilian Michels <[email protected]> wrote: >> >>> Thanks for starting the conversation. The two approaches both look good >>> to me. Probably we want to start with approach #1 for all Runners to be >>> able to support delaying bundles. Flink supports cycles and thus >>> approach #2 would also be applicable and could be used to implement >>> dynamic splitting. >>> >>> -Max >>> >>> On 05.10.20 23:13, Luke Cwik wrote: >>> > Thanks Boyuan, I left a few comments. >>> > >>> > On Mon, Oct 5, 2020 at 11:12 AM Boyuan Zhang <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > Hi team, >>> > >>> > I'm looking at adding self-checkpoint support to portable Flink >>> > runner(BEAM-10940 >>> > <https://issues.apache.org/jira/browse/BEAM-10940>) for both batch >>> > and streaming. I summarized the problem that we want to solve and >>> > proposed 2 potential approaches in this doc >>> > < >>> https://docs.google.com/document/d/1372B7HYxtcUYjZOnOM7OBTfSJ4CyFg_gaPD_NUxWClo/edit?usp=sharing >>> >. >>> > >>> > I want to collect feedback on which approach is preferred and >>> > anything that I have not taken into consideration yet but I should. >>> > Many thanks to all your help! >>> > >>> > Boyuan >>> > >>> >>
