[
https://issues.apache.org/jira/browse/FLINK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418213#comment-17418213
]
Thomas Weise commented on FLINK-24347:
--------------------------------------
Thanks for fixing this. In unbounded executions readers should stay alive
because new partitions/splits can be discovered dynamically.
> KafkaSource cannot checkpoint if the parallelism is higher than the partition
> number
> ------------------------------------------------------------------------------------
>
> Key: FLINK-24347
> URL: https://issues.apache.org/jira/browse/FLINK-24347
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.14.0, 1.13.2
> Reporter: Fabian Paul
> Assignee: Fabian Paul
> Priority: Blocker
> Labels: pull-request-available
>
> The KafkaSourceEnumerator signals all the readers if there are no more splits
> available and the readers shut down. In case the parallelism is higher than
> the partitions of the consumed topic there are not enough partitions to
> distribute and reader subtasks go immediately into FINISHED state.
> Currently, it is not possible to checkpoint if parts of the job are finished.
> (this is lifted once FLIP-147 is by default enabled Flink 1.15+)
> We should only signal to the readers to go into finished if the job is in
> bounded execution mode and keep them idling otherwise.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
