[
https://issues.apache.org/jira/browse/FLINK-38724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040535#comment-18040535
]
yuanfenghu edited comment on FLINK-38724 at 11/25/25 9:01 AM:
--------------------------------------------------------------
Thanks for [~gyfora] quick reply. I think we can limit the max-factor to 0.33
in the autoscaler when we identify some vertex as kafka consumer or veretx
after key by . EVENLY_SPREAD is turned on at the same time ,WDYT?
was (Author: JIRAUSER296932):
Thanks for [~gyfora] quick reply. I think we can limit the max-factor to 0.33
in the autoscaler when we identify some vertex as kafka consumer or veretx
after key by and EVENLY_SPREAD is turned on at the same time, introduced as an
experimental feature ,WDYT?
> Allow per-vertex configuration of scale-down.max-factor to support balanced
> partition consumption
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-38724
> URL: https://issues.apache.org/jira/browse/FLINK-38724
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Affects Versions: kubernetes
> Reporter: yuanfenghu
> Priority: Major
>
> FLINK-36527 introduced the
> job.autoscaler.scaling.key-group.partitions.adjust.mode=EVENLY_SPREAD
> configuration to solve the problem of KeyBy and Kafka
>
> The current default value of job.autoscaler.scale-down.max-factor is 0.6,
> which means that a vertex can only be scaled down to the original parallelism
> in a single scale-down.
>
> Specific scenario:
> - Number of Kafka partitions:4
> - Current parallelism:4
> - Ideal parallelism during trough:2
> - Reduction calculation:4 × 0.6 = 2.4
> - Due to balanced consumption constraints, 2.4 will be adjusted upward to
> 4(must be an integer that evenly allocates 4 partitions)
> - Result: The vertex cannot be scaled down and always remains at a
> parallelism of 4
>
> This violates Autoscaler's goal of reducing resource consumption during low
> times
>
> Provides one of two solutions:
>
> Option 1(Interim Option):
> Adjust the global default job.autoscaler.scale-down.max-factor to a value of
> 0.33 or less to support greater scale-down (for example, from 4 to 2).
>
> Option 2(Recommended Option):
> Added per-vertex configuration that allows you to specify a separate
> max-factor value for a specific vertex, for example:
> job.autoscaler.vertex. <vertex-id>.scale-down.max-factor=0.4
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
