[
https://issues.apache.org/jira/browse/BEAM-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695713#comment-16695713
]
Jozef Vilcek commented on BEAM-6077:
------------------------------------
Ehm, sorry for mixed up names :)
I have a git of a headache to wrap my head around the behavior in
UndoundedSourceWrapper. Important methods seems to be initializeState(),
snapshotState() and open(). Am I correct assuming that snapshotState() should
add to the ListState partitions that current running instance governs and
during the initalizeState() and open(), UnboundedSourceWrapper have access to
all ListState elements from all parallel unbounded sources, so it can pick
other partitions that they were not assigned before. If this is true, it is a
feature of a ListState, because of some global Key?
Anyway, if yes then doing splitting according to MAX_PARALLELISM sounds like a
great idea. Did not notice at fist List `localSplitSources`. I will try it out.
> Make UnboundedSource state rescale friendly
> -------------------------------------------
>
> Key: BEAM-6077
> URL: https://issues.apache.org/jira/browse/BEAM-6077
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Jozef Vilcek
> Assignee: Jozef Vilcek
> Priority: Major
>
> FlinkRunner's UnboundedSourceWrapper currently does not rescale well it's
> state when job changes parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
