[
https://issues.apache.org/jira/browse/FLINK-26358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498532#comment-17498532
]
Shen Zhu edited comment on FLINK-26358 at 2/27/22, 6:38 AM:
------------------------------------------------------------
Hi Fabian,
I did some investigations about this issue and seems the reason is that when
creating JobVertex, it is only using the [max parallelism of start
node|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L770]
for the chained operator.
Therefore, I created a map storing the max parallelism of each operator chain
and use that to set JobVertex, the
[PR|https://github.com/apache/flink/pull/18926] is created here. Would you mind
taking a look when you have a moment?
Thanks!
was (Author: shenzhu0127):
Hi Fabian,
I did some investigations about this issue and seems the reason is that when
creating JobVertex, it is only using the [max parallelism of start
node|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L770]
for the chained operator.
Therefore, I created a map storing the max parallelism of each operator chain
and use that to set JobVertex, the
[PR|https://github.com/apache/flink/pull/18926] is created here. Would you mind
taking a look when you have a moment?(I will mark that PR as "Ready for review"
once all tests passed)
Thanks!
> Operator maxParallelism is lost during chaining
> -----------------------------------------------
>
> Key: FLINK-26358
> URL: https://issues.apache.org/jira/browse/FLINK-26358
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.15.0, 1.14.3
> Reporter: Fabian Paul
> Priority: Critical
> Labels: pull-request-available
>
> During the generation of the JobGraph from the StreamGraph the maxParallelism
> of the chained operators is lost so the maxParallelism written to a snapshot
> might not reflect the real maxParallelism of the operator.
> If a user now unchains the operator it is not possible to restore a snapshot
> anymore because the maxParallelism of the operator and the snapshot do not
> match anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
