[
https://issues.apache.org/jira/browse/FLINK-28054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738109#comment-17738109
]
Alexander Fedulov edited comment on FLINK-28054 at 6/28/23 1:07 PM:
--------------------------------------------------------------------
For reference, in the DataGeneratorSource this was achieved with the
RateLimiterStrategy:
[DataGeneratorPerCheckpoint|https://github.com/apache/flink/blob/873a56361bfd77c828ee743febc9dda2bb044791/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/datagen/DataGeneratorPerCheckpoint.java#L30]
was (Author: afedulov):
For reference, in the DataGeneratorSource this was achieved via the
RateLimiterStrategy:
[DataGeneratorPerCheckpoint|https://github.com/apache/flink/blob/873a56361bfd77c828ee743febc9dda2bb044791/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/datagen/DataGeneratorPerCheckpoint.java#L30]
> Introduce Source API utilities to control records emitted per checkpoint
> ------------------------------------------------------------------------
>
> Key: FLINK-28054
> URL: https://issues.apache.org/jira/browse/FLINK-28054
> Project: Flink
> Issue Type: Sub-task
> Reporter: Alexander Fedulov
> Priority: Major
>
> For some connectors, it is required to verify the content of a checkpoint
> after processing specific records. See [1] for more details. This can be done
> using the SourceFunction like demonstrated here [2]. An abstraction to
> support this use case with the new Source API is required.
> Idea: SourceReader interface has notifyCheckpointComplete() callback, so a
> higher level implementation could be provided to emit records in lockstep
> with checkpoints.
>
> [1]https://lists.apache.org/thread/9kfppzbq7r2gk5mhykc7m786sv0fc8j2
> [2]https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
