[
https://issues.apache.org/jira/browse/FLINK-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336905#comment-17336905
]
Flink Jira Bot commented on FLINK-2824:
---------------------------------------
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Iteration feedback partitioning does not work as expected
> ---------------------------------------------------------
>
> Key: FLINK-2824
> URL: https://issues.apache.org/jira/browse/FLINK-2824
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
> Affects Versions: 0.10.0
> Reporter: Gyula Fora
> Priority: Major
> Labels: stale-major
>
> Iteration feedback partitioning is not handled transparently and can cause
> serious issues if the user does not know the specific implementation details
> of streaming iterations (which is not a realistic expectation).
> Example:
> IterativeStream it = ... (parallelism 1)
> DataStream mapped = it.map(...) (parallelism 2)
> // this does not work as the feedback has parallelism 2 != 1
> // it.closeWith(mapped.partitionByHash(someField))
> // so we need rebalance the data
> it.closeWith(mapped.map(NoOpMap).setParallelism(1).partitionByHash(someField))
> This program will execute but the feedback will not be partitioned by hash to
> the mapper instances:
> The partitioning will be set from the noOpMap to the iteration sink which has
> parallelism different from the mapper (1 vs 2) and then the iteration source
> forwards the element to the mapper (always to 0).
> So the problem is basically that the iteration source/sink pair gets the
> parallelism of the input stream (p=1) not the head operator (p = 2) which
> leads to incorrect partitioning.
> Workaround:
> Set input parallelism to the same as the head operator
> Suggested solution:
> The iteration construction should be reworked to set the parallelism of the
> source/sink to the parallelism of the head operator (and validate that all
> heads have the same parallelism)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
