[
https://issues.apache.org/jira/browse/FLINK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-32872:
----------------------------------
Description:
*Problem*
Currently, when the no partitioner is specified, FORWARD partitioner is used if
the parallelism of upstream and downstream operator matches, REBALANCE
partitioner used otherwise. However, this behavior is not configurable and can
be undesirable in certain cases:
# REBALANCE partitioner will create an all-to-all connection between upstream
and downstream operators and consume a lot of extra CPU and memory resources
when the parallelism is high in pipelining mode and RESCALE partitioner is
desirable in this case.
# For Flink SQL jobs, users cannot specify the partitioner directly so far.
And for DataStream jobs, users may not want to explicitly set the partitioner
everywhere.
*Proposal*
Add an option to control the default partitioner when the parallelism of
upstream and downstream operator does not match. The option can have the name
"pipeline.default-partitioner-with-unmatched-parallelism" with REBALANCE as the
default value.
was:
*Problem*
Currently, when the no partitioner is specified, FORWARD partitioner is used if
the parallelism of upstream and downstream operator matches, REBALANCE
partitioner used otherwise. However, this behavior is not configurable and can
be undesirable in certain cases:
# REBALANCE partitioner will create an all-to-all connection between upstream
and downstream operators and consume a lot of extra CPU and memory resources
when the parallelism is high in pipelining mode and RESCALE partitioner is
desirable in this case.
# For Flink SQL jobs, users cannot specify the partitioner directly so far.
And for DataStream jobs, users may not want to explicitly set the partitioner
everywhere.
*Proposal*
Add an option to control the default partitioner when the parallelism of
upstream and downstream operator does not match. The option can have the name
"pipeline.default-partioner-with-unmatched-parallelism" with REBALANCE as the
default value.
> Add option to control the default partitioner when the parallelism of
> upstream and downstream operator does not match
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-32872
> URL: https://issues.apache.org/jira/browse/FLINK-32872
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration
> Affects Versions: 1.17.0
> Reporter: Zhanghao Chen
> Priority: Major
>
> *Problem*
> Currently, when the no partitioner is specified, FORWARD partitioner is used
> if the parallelism of upstream and downstream operator matches, REBALANCE
> partitioner used otherwise. However, this behavior is not configurable and
> can be undesirable in certain cases:
> # REBALANCE partitioner will create an all-to-all connection between
> upstream and downstream operators and consume a lot of extra CPU and memory
> resources when the parallelism is high in pipelining mode and RESCALE
> partitioner is desirable in this case.
> # For Flink SQL jobs, users cannot specify the partitioner directly so far.
> And for DataStream jobs, users may not want to explicitly set the partitioner
> everywhere.
> *Proposal*
> Add an option to control the default partitioner when the parallelism of
> upstream and downstream operator does not match. The option can have the name
> "pipeline.default-partitioner-with-unmatched-parallelism" with REBALANCE as
> the default value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
