I assume you are talking about the checkpointing in the feedback package? StateFun only relies on Flink checkpointing for fault tolerance. All state is stored in standard checkpoint / savepoints and can be used to restore from failure, upgrade a job, rescale, etc. Just like any other snapshot.
StateFun internally relies on a form of streaming iteration. The feedback package contains the code for that iteration which sends records back upstream. To ensure exactly-once semantics, in-flight records on the feedback edge need to be stored in the snapshot. It does so through Flinks raw keyed state abstraction. This is the same thing state backends use under the hood to snapshot out your keyed state in any rich function. Raw keyed state has been a stable part of Flinks API forever, but is rarely seen in user code. I often describe the statefun runtime to people as the world's most complex datastream application; you will find advanced usage patterns most people have not seen, but nothing that isn't well supported by Flink. Seth On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson <chr...@kth.se> wrote: > Hi, > > > I am doing some experiments with StateFun and different Flink state > backends. I was looking through the StateFun source code and saw that it is > doing some own logic when it comes to checkpointing. My initial > understanding was that checkpointing was implicitly leveraged with Flink. I > did not find a lot of documentation about this, so I am asking if you could > provide a description of how checkpointing works in StateFun, and how it is > different from regular Flink checkpointing? > > > Thanks in advance, > > Christopher Gustafson >
