Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi Stephan, Thank you so much for all these practical suggestions. I agree that hybrid batch/streaming execution is an awesome idea and that no checkpoint could be successful during the batch phase. However, the checkpoint could fail frequently because of the large batch size, which might be

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Before jumping into the designs of other mechanisms, can we clarify whether adjusting checkpointing is really the right approach here? What about storing empty state in the checkpoints, so that any recovery simply does a full replay of the input. Regarding the hybrid batch/streaming execution:

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi Stephan, Thank you so much for replying and suggesting! Follow by your question, I would give some explanation and new thoughts. (1) More detailed info about CDC use case. In the previous design of FLINK-CDC, they would start a full-table scanning at the beginning by holding a read-write

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi! Thanks for writing this FLIP, and interesting idea. I would like to understand a bit better why exactly we need this, and what our alternative options are. My main concerns are the following: *(1) Can we achieve this without changing the checkpointing mechanism?* The checkpoint mechanism

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi Piotrek, Thanks for your feedback! 1. Why not ExternallyInducedSourceReader/ExternallyInducedSource? a. The `org.apache.flink.api.connector.source.ExternallyInducedSourceReader` and `org.apache.flink.api.connector.source.ExternallyInducedSource` seems like playing the role of checkpoint

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi Senhong, Thanks for the proposal. I have a couple of questions. Have you seen `org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource` (for the legacy SourceFunction) and `org.apache.flink.api.connector.source.ExternallyInducedSourceReader` (for FLIP-27) interfaces? They work the

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Here is some brief context about the new feature. 1. Actively checkpoint rejecting by the operator. Follow by the current checkpoint mechanism, one more preliminary step is added to help the operator determine that if it is able to take snapshots. The preliminary step is a new API provided to the

[DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

Hi guys, We would like to start a discussion on the new FLIP about rejecting checkpoints on the operator level. The basic idea is to allow the operator to reject a checkpoint when it is not under a proper situation and returning a proper failure reason.