[
https://issues.apache.org/jira/browse/BEAM-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689405#comment-16689405
]
Jozef Vilcek commented on BEAM-6077:
------------------------------------
This is a big bigger change than I estimated by quick scan of code. State
snapshot in the wrapper relies on `CheckpointMark` which bears not much of
information. Also, I did not notice that each reader returns single
CheckpointMark and not the list, so final ListState can not be easily re-split
which I though as first possible solution.
So, if:
a) want to redistribute the state from list by re-splitting it, the question
would be how to flatten CheckpointMark
b) want to use Keyed state, how to get Key from reader or CheckpointMark and
again, make the mark more granular
Both seems to be invasive and enter beam core. I hope there is option c) I can
not see as Beam rookie :)
Any thoughts much appreciated
> Make UnboundedSource state rescale friendly
> -------------------------------------------
>
> Key: BEAM-6077
> URL: https://issues.apache.org/jira/browse/BEAM-6077
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Jozef Vilcek
> Assignee: Jozef Vilcek
> Priority: Major
>
> FlinkRunner's UnboundedSourceWrapper currently does not rescale well it's
> state when job changes parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
