Zhanghao Chen created FLINK-32872: ------------------------------------- Summary: Add option to control the default partitioner when the parallelism of upstream and downstream operator does not match Key: FLINK-32872 URL: https://issues.apache.org/jira/browse/FLINK-32872 Project: Flink Issue Type: New Feature Components: Runtime / Configuration Affects Versions: 1.17.0 Reporter: Zhanghao Chen
*Problem* Currently, when the no partitioner is specified, FORWARD partitioner is used if the parallelism of upstream and downstream operator matches, REBALANCE partitioner used otherwise. However, this behavior is not configurable and can be undesirable in certain cases: # REBALANCE partitioner will create an all-to-all connection between upstream and downstream operators and consume a lot of extra CPU and memory resources when the parallelism is high in pipelining mode and RESCALE partitioner is desirable in this case. # For Flink SQL jobs, users cannot specify the partitioner directly so far. And for DataStream jobs, users may not want to explicitly set the partitioner everywhere. *Proposal* Add an option to control the default partitioner when the parallelism of upstream and downstream operator does not match. The option can have the name "pipeline.default-partioner-with-unmatched-parallelism" with REBALANCE as the default value. -- This message was sent by Atlassian Jira (v8.20.10#820010)