Re: Checkpointing in StateFun
There's a passing reference to it in the 1.2 docs but we removed it to discourage use. https://nightlies.apache.org/flink/flink-docs-release-1.2/dev/stream/state.html#raw-and-managed-state On Sat, Mar 12, 2022 at 2:50 AM Christopher Gustafson wrote: > Hi, > > > Thanks for the explanation, I had not heard about the Raw State before, > and it sounds like that is not surprising if it is rarely used, and doesn't > seem to be included in the latest versions of the documentation. Is there > perhaps some older version of the documentation you could point me to where > I can see how Raw state can be used in Flink? > > > Best Regards, > > Christopher Gustafson > -- > *Från:* Seth Wiesman > *Skickat:* den 11 mars 2022 17:57:21 > *Till:* Christopher Gustafson > *Kopia:* user@flink.apache.org > *Ämne:* Re: Checkpointing in StateFun > > I assume you are talking about the checkpointing in the feedback package? > > StateFun only relies on Flink checkpointing for fault tolerance. All state > is stored in standard checkpoint / savepoints and can be used to restore > from failure, upgrade a job, rescale, etc. Just like any other snapshot. > > StateFun internally relies on a form of streaming iteration. The feedback > package contains the code for that iteration which sends records back > upstream. To ensure exactly-once semantics, in-flight records on the > feedback edge need to be stored in the snapshot. It does so through Flinks > raw keyed state abstraction. This is the same thing state backends use > under the hood to snapshot out your keyed state in any rich function. > > Raw keyed state has been a stable part of Flinks API forever, but is > rarely seen in user code. I often describe the statefun runtime to people > as the world's most complex datastream application; you will find advanced > usage patterns most people have not seen, but nothing that isn't well > supported by Flink. > > Seth > > On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson > wrote: > >> Hi, >> >> >> I am doing some experiments with StateFun and different Flink state >> backends. I was looking through the StateFun source code and saw that it is >> doing some own logic when it comes to checkpointing. My initial >> understanding was that checkpointing was implicitly leveraged with Flink. I >> did not find a lot of documentation about this, so I am asking if you could >> provide a description of how checkpointing works in StateFun, and how it is >> different from regular Flink checkpointing? >> >> >> Thanks in advance, >> >> Christopher Gustafson >> >
SV: Checkpointing in StateFun
Hi, Thanks for the explanation, I had not heard about the Raw State before, and it sounds like that is not surprising if it is rarely used, and doesn't seem to be included in the latest versions of the documentation. Is there perhaps some older version of the documentation you could point me to where I can see how Raw state can be used in Flink? Best Regards, Christopher Gustafson Från: Seth Wiesman Skickat: den 11 mars 2022 17:57:21 Till: Christopher Gustafson Kopia: user@flink.apache.org Ämne: Re: Checkpointing in StateFun I assume you are talking about the checkpointing in the feedback package? StateFun only relies on Flink checkpointing for fault tolerance. All state is stored in standard checkpoint / savepoints and can be used to restore from failure, upgrade a job, rescale, etc. Just like any other snapshot. StateFun internally relies on a form of streaming iteration. The feedback package contains the code for that iteration which sends records back upstream. To ensure exactly-once semantics, in-flight records on the feedback edge need to be stored in the snapshot. It does so through Flinks raw keyed state abstraction. This is the same thing state backends use under the hood to snapshot out your keyed state in any rich function. Raw keyed state has been a stable part of Flinks API forever, but is rarely seen in user code. I often describe the statefun runtime to people as the world's most complex datastream application; you will find advanced usage patterns most people have not seen, but nothing that isn't well supported by Flink. Seth On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson mailto:chr...@kth.se>> wrote: Hi, I am doing some experiments with StateFun and different Flink state backends. I was looking through the StateFun source code and saw that it is doing some own logic when it comes to checkpointing. My initial understanding was that checkpointing was implicitly leveraged with Flink. I did not find a lot of documentation about this, so I am asking if you could provide a description of how checkpointing works in StateFun, and how it is different from regular Flink checkpointing? Thanks in advance, Christopher Gustafson
Re: Checkpointing in StateFun
I assume you are talking about the checkpointing in the feedback package? StateFun only relies on Flink checkpointing for fault tolerance. All state is stored in standard checkpoint / savepoints and can be used to restore from failure, upgrade a job, rescale, etc. Just like any other snapshot. StateFun internally relies on a form of streaming iteration. The feedback package contains the code for that iteration which sends records back upstream. To ensure exactly-once semantics, in-flight records on the feedback edge need to be stored in the snapshot. It does so through Flinks raw keyed state abstraction. This is the same thing state backends use under the hood to snapshot out your keyed state in any rich function. Raw keyed state has been a stable part of Flinks API forever, but is rarely seen in user code. I often describe the statefun runtime to people as the world's most complex datastream application; you will find advanced usage patterns most people have not seen, but nothing that isn't well supported by Flink. Seth On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson wrote: > Hi, > > > I am doing some experiments with StateFun and different Flink state > backends. I was looking through the StateFun source code and saw that it is > doing some own logic when it comes to checkpointing. My initial > understanding was that checkpointing was implicitly leveraged with Flink. I > did not find a lot of documentation about this, so I am asking if you could > provide a description of how checkpointing works in StateFun, and how it is > different from regular Flink checkpointing? > > > Thanks in advance, > > Christopher Gustafson >
Checkpointing in StateFun
Hi, I am doing some experiments with StateFun and different Flink state backends. I was looking through the StateFun source code and saw that it is doing some own logic when it comes to checkpointing. My initial understanding was that checkpointing was implicitly leveraged with Flink. I did not find a lot of documentation about this, so I am asking if you could provide a description of how checkpointing works in StateFun, and how it is different from regular Flink checkpointing? Thanks in advance, Christopher Gustafson
