[
https://issues.apache.org/jira/browse/FLINK-36531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899228#comment-17899228
]
Sai Sharath Dandi commented on FLINK-36531:
-------------------------------------------
My understanding is that FLIP-461 accumulates the scaling actions from
autoscale on the scheduler side, so it will not use an outdated action. I think
the improvement to rescale early than delay the scaling till the next
checkpoint(in case of in-frequent checkpoints) could be better handled on the
scheduler side rather than autoscaler. Overall though, I feel there is a lack
of clear separation of concerns across autoscaler and scheduler. Autoscaler
isn't "aware" of what's going on, on the scheduler side, which causes the
scaling actions to be delayed.
> AutoScaler needs to consider the lag from last checkpoint
> ---------------------------------------------------------
>
> Key: FLINK-36531
> URL: https://issues.apache.org/jira/browse/FLINK-36531
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Sai Sharath Dandi
> Priority: Major
>
> Autoscaler computes the target processing capacity as
> [below|https://sg.uberinternal.com/code.uber.internal/uber-code/[email protected]/-/blob/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/AutoScalerUtils.java?L47]
> // Target = LAG/CATCH_UP + INPUT_RATE*RESTART/CATCH_UP +
> INPUT_RATE/TARGET_UTIL
>
> During the scaling action, the autoscaler will restart the job from the last
> successful checkpoint, we need to include the number of processed records
> since last successful checkpoint as part of the lag as those records will be
> replayed after scaling. This is particularly important for jobs with long
> checkpoint intervals and large state as there could be a significant
> difference between the realtime lag and the lag from the checkpoint
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
