[
https://issues.apache.org/jira/browse/BEAM-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701665#comment-16701665
]
Maximilian Michels commented on BEAM-6077:
------------------------------------------
Your observation concerning Flink is correct. The partitions are discovered
during runtime and assigned using the subtask index.
I don't understand the problem concerning Kafka, it seems to pre-assign the
splits when creating UnboundedKafakSource. Looking at the source, it seems to
never create more splits than the partition size, but I'm not that familiar
with Beam's Kafka connector.
> Make UnboundedSource state rescale friendly
> -------------------------------------------
>
> Key: BEAM-6077
> URL: https://issues.apache.org/jira/browse/BEAM-6077
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Jozef Vilcek
> Assignee: Jozef Vilcek
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> FlinkRunner's UnboundedSourceWrapper currently does not rescale well it's
> state when job changes parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
