Hi all! I would like to start a discussion on state catch-up using BATCH execution mode [1]
FLIP-134 [2] introduced BATCH execution mode as a way to execute more efficiently DataStream API jobs on bounded inputs. As even stated in the documentation [3]: > One obvious outlier is when you want to use a bounded job to bootstrap some job state that you > then want to use in an unbounded job. For example, by running a bounded job using > STREAMING mode, taking a savepoint, and then restoring that savepoint on an unbounded job. > This is a very specific use case and one that might soon become obsolete when we allow > producing a savepoint as additional output of a BATCH execution job. producing a savepoint from BATCH execution job has always been planned as a future extension. FLIP-605 [1] was created to implement this. The idea is that instead of waiting sometimes hours to process a long backlog of records using STREAMING execution mode, users could submit their jobs in BATCH execution mode, create a savepoint at the end, and then recover their jobs from that savepoint and continue running them in STREAMING. Regardless if those jobs are using DataStream API or Flink Streaming SQL/Table API. For more information please take a look into the FLIP document itself [1]. Disclosure, in Confluent/IBM we have a working early access version of this feature, hence we already have a pretty good understanding of what needs to be done to make it work and what are the expected results. Best, Piotrek [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/446071315/FLIP-605+State+catch-up+of+streaming+jobs+state+using+BATCH+execution+mode [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/158871522/FLIP-134+Batch+execution+for+the+DataStream+API [3] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/#when-canshould-i-use-batch-execution-mode
