JingsongLi commented on issue #9803: [FLINK-14265][table-planner-blink] Don't use ContinuousFileReaderOperator to support multiple paths URL: https://github.com/apache/flink/pull/9803#issuecomment-545755404 > ``` > * <p><b>NOTES ON CHECKPOINTING: </b> In the case of a {@link FileInputFormat}, the source > * (which executes the {@link ContinuousFileMonitoringFunction}) monitors the path, creates the > * {@link org.apache.flink.core.fs.FileInputSplit FileInputSplits} to be processed, forwards > * them to the downstream {@link ContinuousFileReaderOperator} to read the actual data, and exits, > * without waiting for the readers to finish reading. This implies that no more checkpoint > * barriers are going to be forwarded after the source exits, thus having no checkpoints. > ``` > > It seems that there are some benefit when use `ContinuousFileMonitoringFunction` in streaming mode. But the cost is that we can not use multi-paths. After discuss with @wuchong offline, I will use `ContinuousFileMonitoringFunction` in streaming-mode (require streaming exactly-once) and use `InputFormatSourceFunction` in batch-mode (do not need care about checkpoints).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
