Rui Fan created FLINK-36535:
-------------------------------
Summary: Optimize the scale down logic based on historical
parallelism
Key: FLINK-36535
URL: https://issues.apache.org/jira/browse/FLINK-36535
Project: Flink
Issue Type: Improvement
Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
This is a follow-up to FLINK-36018 . FLINK-36018 supported the lazy scale down
to avoid frequent rescaling.
h1. Background
Some critical Flink jobs need to scale up in time, but only scale down on a
daily basis. In other words, Flink users do not want Flink jobs to be scaled
down multiple times within 24 hours, and the jobs run at the same parallelism
as during the peak hours of each day.
Note: Users hope to scale down only happens when the parallelism during peak
hours is still a waste of resources. This is a trade-off between downtime and
resource waste for a critical job.
h1. Current solution
In general, this requirement could be met after setting{color:#de350b}
job.autoscaler.scale-down.interval= 24 hour{color}. For example, the vertex1
runs with parallelism=100, and the following is the parallelism that the
autoscaler recommends for vertex1:
* 100 (2024-10-13 20:00:00, peak hour)
* 90 (2024-10-13 21:00:00, trigger delayed scale down)
* 80 (2024-10-13 22:00:00)
* 70 (2024-10-14 00:00:00)
* 60 (2024-10-14 01:00:00)
* 50 (2024-10-14 02:00:00)
* 40 (2024-10-14 04:00:00)
* 50 (2024-10-14 06:00:00)
* 60 (2024-10-14 08:00:00)
* ...
* 90 (2024-10-14 19:00:00)
* 100 (2024-10-14 20:00:00, peak hour, the delayed scale down is canceled)
All recommended parallelism are delayed, and the recommended parallelism is
backed to 100 within 24 hours. So the scale down request is canceled.
It means if the recommended parallelism for vertex1 during peak hours is 100
every day, this vertex1 never be scaled down and scaled up. It is very friendly
to critical jobs, and reducing the scale frequency can greatly reduce the
downtime.
h1. Some scenarios do not work as expected
When scale down occurs, the latest recommended parallelism is used, not the
highest parallelism in the last 24 hours. For example:
* 100 (2024-10-13 20:00:00, peak hour)
* 100 (2024-10-13 21:00:00, peak hour)
* 80 (2024-10-13 22:00:00, trigger delayed scale down)
* 60 (2024-10-14 01:00:00)
* 40 (2024-10-14 04:00:00)
* 60 (2024-10-14 08:00:00)
* ...
* 90 (2024-10-14 20:00:00, peak hour)
* 90 (2024-10-14 21:00:00, peak hour)
* 70 (2024-10-14 22:00:00, scale down happens)
In this case, the peak traffic on 2024-10-14 is lower than the peak traffic on
2024-10-13. Therefore, the parallelism on 2024-10-14 is 90, which is lower than
the parallelism of 100 during the peak period on 2024-10-13.
So the delayed scale down happens at 2024-10-14 22:00:00, and the scaled
parallelism is 70.
{color:#de350b}Unexpected logic{color}:
* It's better to use the 90 as the new parallelism instead of 70.
* Generally, {color:#de350b}it's better to use the highest parallelism in the
last 24 hours(job.autoscaler.scale-down.interval) as the new parallelism{color}
instead of the latest parallelism.
Reason:
* When we use the latest parallelism as the new parallelism, there is a high
probability that a scale-up will be needed in the next 24 hours.
* Also, if job.autoscaler.scale-down.interval is 36 hours, when scale down
occurs, it may be a low-peak period. 40 or 50 may be the latest parallelism,
there is a high probability that a scale-up will be needed in the next 24
hours.
h1. Solution:
h2. Solution1: introducing an option
Introducing an option to control whether the highest parallelism within the
job.autoscaler.scale-down.interval period or the latest parallelism is applied.
h2. Solution2: do not introduce an option
* When job.autoscaler.scale-down.interval >= 24 hours, using the highest
parallelism within the job.autoscaler.scale-down.interval period.
** We could assume that the user wants the job to achieve the goals I
mentioned at background part when job.autoscaler.scale-down.interval >= 24
hours.
* Otherwise, using the latest parallelism.
** We could assume that the user wants to scale down within a day.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
