[
https://issues.apache.org/jira/browse/FLINK-28054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28054:
--------------------------------------
Description:
For some connectors, it is required to verify the content of a checkpoint after
processing specific records. See [1] for more details. This can be done using
the SourceFunction like demonstrated here [2]. An abstraction to support this
use case with the new Source API is required.
Idea: SourceReader interface has notifyCheckpointComplete() callback, so a
higher level implementation could be provided to emit records in lockstep with
checkpoints.
[1]https://lists.apache.org/thread/9kfppzbq7r2gk5mhykc7m786sv0fc8j2
[2]https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java
was:
Some connectors require verifying the content of a checkpoint after processing
specific records. See [1] for more details. This can be done using the
SourceFunction like demonstrated here [2]. An abstraction to support this use
case with the new Source API is required.
Idea: SourceReader interface has notifyCheckpointComplete() callback, so a
higher level implementation could be provided to emit records in lockstep with
checkpoints.
[1]https://lists.apache.org/thread/9kfppzbq7r2gk5mhykc7m786sv0fc8j2
[2]https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java
> Introduce Source API utilities to control records emitted per checkpoint
> ------------------------------------------------------------------------
>
> Key: FLINK-28054
> URL: https://issues.apache.org/jira/browse/FLINK-28054
> Project: Flink
> Issue Type: Sub-task
> Reporter: Alexander Fedulov
> Priority: Major
>
> For some connectors, it is required to verify the content of a checkpoint
> after processing specific records. See [1] for more details. This can be done
> using the SourceFunction like demonstrated here [2]. An abstraction to
> support this use case with the new Source API is required.
> Idea: SourceReader interface has notifyCheckpointComplete() callback, so a
> higher level implementation could be provided to emit records in lockstep
> with checkpoints.
>
> [1]https://lists.apache.org/thread/9kfppzbq7r2gk5mhykc7m786sv0fc8j2
> [2]https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
