Hi, Piotrek. Thanks a lot for putting this FLIP together. After modifying a streaming job, having to restart it statelessly often comes at a considerable time and resource cost for users. So big +1 from me. I've gone through the FLIP and it looks good to me overall. I just have a few questions I'd like to check with you: 1. If the batch phase produces a savepoint in the standard format, my understanding is that the streaming phase could restore from it directly via `state.savepoints.dir`. If that's true, would we still need to set `execution.state-catch-up.mode: BATCH` in the streaming phase, or am I missing something here? 2. Regarding sources, I'm wondering whether it might have an interface to indicate whether a source supports the bounded-to-unbounded transition. My thinking is that existing bounded sources in current connectors don't record their own state when they finish, so the subsequent streaming phase would have nothing to pick up from. Curious whether you've already considered how to handle this. 3. I also have another rather rough idea around execution in the batch phase, and I'd be very interested in your thoughts on it. For very large full datasets, a single failure midway can invalidate hours of earlier batch computation. Even with fine-grained restarts in the batch phase down the road, a single full-scale run would still need to retain the blocking shuffle partitions for the entire backlog until they're consumed,so both TM disk pressure and the recomputation cost would scale with the full dataset. The idea would be to keep catch-up as a single job, but internally split the historical data into N shards(or N consecutive segments of data) that are consumed sequentially — producing a complete checkpoint as each finishes, and producing the save point handed over to the streaming phase when the last one completes. This feels somewhat in the spirit of what FLIP-309[1] describes, where checkpointing isn't disabled during backlog processing but simply happens at a lower frequency. That said, I may well be overlooking some complexity here, so please feel free to point out anything I've missed. Thanks again for the great work! Looking forward to your thoughts.
[1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/255069517/FLIP-309+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog. -- Best！ Xuyang At 2026-08-07 21:53:16, "Piotr Nowojski" <[email protected]> wrote: >Hi all! > >I would like to start a discussion on state catch-up using BATCH execution >mode [1] > >FLIP-134 [2] introduced BATCH execution mode as a way to execute more >efficiently DataStream API jobs on bounded inputs. As even stated in the >documentation [3]: > >> One obvious outlier is when you want to use a bounded job to bootstrap >some job state that you >> then want to use in an unbounded job. For example, by running a bounded >job using >> STREAMING mode, taking a savepoint, and then restoring that savepoint on >an unbounded job. >> This is a very specific use case and one that might soon become obsolete >when we allow >> producing a savepoint as additional output of a BATCH execution job. > >producing a savepoint from BATCH execution job has always been planned as a >future extension. FLIP-605 [1] was created to implement this. > >The idea is that instead of waiting sometimes hours to process a long >backlog of records using STREAMING execution mode, users could submit their >jobs in BATCH execution mode, create a savepoint at the end, and then >recover their jobs from that savepoint and continue running them in >STREAMING. Regardless if those jobs are using DataStream API or Flink >Streaming SQL/Table API. > >For more information please take a look into the FLIP document itself [1]. > >Disclosure, in Confluent/IBM we have a working early access version of this >feature, hence we already have a pretty good understanding of what needs to >be done to make it work and what are the expected results. > >Best, >Piotrek > >[1] >https://cwiki.apache.org/confluence/spaces/FLINK/pages/446071315/FLIP-605+State+catch-up+of+streaming+jobs+state+using+BATCH+execution+mode >[2] >https://cwiki.apache.org/confluence/spaces/FLINK/pages/158871522/FLIP-134+Batch+execution+for+the+DataStream+API >[3] >https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/#when-canshould-i-use-batch-execution-mode
