echauchot commented on code in PR #22985:
URL: https://github.com/apache/flink/pull/22985#discussion_r1321602343
##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/Executing.java:
##########
@@ -124,17 +154,33 @@ private void handleDeploymentFailure(ExecutionVertex
executionVertex, JobExcepti
@Override
public void onNewResourcesAvailable() {
- maybeRescale();
+ rescaleWhenCooldownPeriodIsOver();
}
@Override
public void onNewResourceRequirements() {
- maybeRescale();
+ rescaleWhenCooldownPeriodIsOver();
}
private void maybeRescale() {
- if (context.shouldRescale(getExecutionGraph())) {
- getLogger().info("Can change the parallelism of job. Restarting
job.");
+ final Duration timeSinceLastRescale = timeSinceLastRescale();
+ rescaleScheduled = false;
+ final boolean shouldForceRescale =
+ (scalingIntervalMax != null)
+ && (timeSinceLastRescale.compareTo(scalingIntervalMax)
> 0)
+ && (lastRescale != Instant.EPOCH); // initial rescale
is not forced
+ if (shouldForceRescale || context.shouldRescale(getExecutionGraph())) {
+ if (shouldForceRescale) {
+ getLogger()
+ .info(
+ "Time since last rescale ({}) > {} ({}).
Force-changing the parallelism of the job. Restarting the job.",
+ timeSinceLastRescale,
+
JobManagerOptions.SCHEDULER_SCALING_INTERVAL_MAX.key(),
+ scalingIntervalMax);
+ } else {
+ getLogger().info("Can change the parallelism of the job.
Restarting the job.");
+ }
+ lastRescale = Instant.now();
context.goToRestarting(
getExecutionGraph(),
Review Comment:
> Do you mean we force trigger a rescale when new resources comes and the
current time > lastRescaleTime + scalingIntervalMax?
yes, that is what the community agreed on in the FLIP.
> If so, I think the logic related to scalingIntervalMax should be moved
into maybeRescale.
This logic is already in maybeRescale() method
> onNewResourceRequirements also call the rescaleWhenCooldownPeriodIsOver,
when user changed the parallelism; maybeRescale will force trigger a rescale as
well. right?
yes `onNewResourceRequirements` is ultimately called by a rest endpoint to
react to use action on the web ui so it will call maybeRescale in the end but
that would not always mean a rescale: if timeSinceLastRescale <
scalingIntervalMax and minimal resources are not met then there will be no
rescale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
