[
https://issues.apache.org/jira/browse/FLINK-32124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723796#comment-17723796
]
Gyula Fora commented on FLINK-32124:
------------------------------------
This is related to https://issues.apache.org/jira/browse/FLINK-32124
But I think currently we actually consider partition balance. We set the max
parallelism to the number of partitions and only select parallelisms that are
divisors of this (so there is always balance)
> Add option to enable partition alignment for sources
> ----------------------------------------------------
>
> Key: FLINK-32124
> URL: https://issues.apache.org/jira/browse/FLINK-32124
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Zhanghao Chen
> Priority: Major
>
> Currently, autoscaler did not consider balancing partitions among source
> tasks. In our production env, partition skew has proven to be a severe
> problem for many jobs. Especially in a job topology with all forward or
> rescale shuffles, partition skew on the source side can further lead to data
> imbalance in later operators.
> We should add an option to enable partition alignment for sources for that,
> but making it disabled by default as this has a side effect in that partition
> usu. has limited factors and enabling alignment will greatly limit our
> scaling choices. Also, if data among partitions are imbalanced in the first
> place, partition alignment won't help as well (this is not a common case
> inside our company though).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
