[
https://issues.apache.org/jira/browse/FLINK-31996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719216#comment-17719216
]
Chesnay Schepler edited comment on FLINK-31996 at 5/4/23 8:48 AM:
------------------------------------------------------------------
??So I would expect this to not be a breaking change.??
We need an opt-in flag because this change may break existing chains, which can
result in certain types being serialized that previously weren't. This can
affect performance and savepoint compatibility.
E.g., if you have a job that is one big chain then your super special custom
type doesn't go through Kryo. Users may also rely on the same object being
passed through the job.
was (Author: zentol):
?? So I would expect this to not be a breaking change.??
We need an opt-in flag because this change may break existing chains, which can
result in certain types being serialized that previously weren't. This can
affect performance and savepoint compatibility.
E.g., if you have a job that is one big chain then your super special custom
type doesn't go through Kryo. Users may also rely on the same object being
passed through the job.
> Chaining operators with different max parallelism prevents rescaling
> --------------------------------------------------------------------
>
> Key: FLINK-31996
> URL: https://issues.apache.org/jira/browse/FLINK-31996
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Reporter: David Morávek
> Priority: Major
>
> We might chain operators with different max parallelism together if they are
> set to have the same parallelism initially.
> When we decide to rescale the JobGraph vertices (using AdaptiveScheduler),
> we're gapped by the lowest maxParallelism of the operator chain. This is
> especially visible with things like CollectSink, TwoPhaseCommitSink, CDC, and
> a GlobalCommiter with maxParallelism set to 1.
>
> An obvious solution would be to prevent the chaining of operators with
> different maxParallelism, but we need to double-check this doesn't introduce
> a breaking change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
