[
https://issues.apache.org/jira/browse/FLINK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762051#comment-17762051
]
Weihua Hu commented on FLINK-32872:
-----------------------------------
[~Zhanghao Chen] Thanks for bring this. We need a FLIP and public discussion in
the dev ML to adding a new config options.
> Add option to control the default partitioner when the parallelism of
> upstream and downstream operator does not match
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-32872
> URL: https://issues.apache.org/jira/browse/FLINK-32872
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration
> Affects Versions: 1.17.0
> Reporter: Zhanghao Chen
> Priority: Major
>
> *Problem*
> Currently, when the no partitioner is specified, FORWARD partitioner is used
> if the parallelism of upstream and downstream operator matches, REBALANCE
> partitioner used otherwise. However, this behavior is not configurable and
> can be undesirable in certain cases:
> # REBALANCE partitioner will create an all-to-all connection between
> upstream and downstream operators and consume a lot of extra CPU and memory
> resources when the parallelism is high in pipelining mode and RESCALE
> partitioner is desirable in this case.
> # For Flink SQL jobs, users cannot specify the partitioner directly so far.
> And for DataStream jobs, users may not want to explicitly set the partitioner
> everywhere.
> *Proposal*
> Add an option to control the default partitioner when the parallelism of
> upstream and downstream operator does not match. The option can have the name
> "pipeline.default-partitioner-with-unmatched-parallelism" with REBALANCE as
> the default value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
