Sergey Kononov created FLINK-40800:
--------------------------------------
Summary: When scaling in Flink is performed Autoscaler decision is
not applied anymore
Key: FLINK-40800
URL: https://issues.apache.org/jira/browse/FLINK-40800
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Affects Versions: 1.16.1
Environment: Real kubernetes cluster with Flink Kubernetes Operator
1.16.1 ({{{}ghcr.io/apache/flink-kubernetes-operator:1c895a3{}}}).
FlinkDeployment with JobSpec ({{{}flink:1.20.5{}}}) running WindowJoin example.
Reporter: Sergey Kononov
I encountered a problem on a Kubernetes cluster with Operator with Autoscaler
enabled. If user scaled a job with {{jobmanager.scheduler: adaptive}} using
Flink Web UI (or REST) Autoscaler can't apply its scaling decision anymore. In
logs user can see only repeating SCALINGREPORT like this:
{noformat}
o.a.f.k.o.l.AuditUtils [INFO ][fko/test-windowjoin-autoscaler] >>>
Event[Job] | Info | SCALINGREPORT | Scaling execution enabled, begin
scaling vertices:{ Vertex ID 17e79f7e136f9548649fb1b735f40eae | Parallelism 10
-> 12 | Processing capacity 1501097.47 -> 1801316.96 | Target data rate
1490558.47}{noformat}
But no in-place scaling trigger after that. It doesn't looks like an intended
behavior.
My investigation showed that it happens because parallelism overrides evaluated
by Autoscaler stay the same and the resource reconciler doesn't see a
difference in specs and skips a scaling phase. One needs to redeploy the job or
apply {{autoscalerResetNonce}} (that leads to a job redeployment) to force
scaling by Autoscaler. That may be undesired. Or one can wait when Autoscaler
evaluates a different desired parallelism then reconciler will see a diff in
specs and apply the parallelism overrides.
I propose a simple solution for this. One can add a scaling counter to JobSpec.
Whenever Autoscaler evaluates a new scaling decision it increments it. This way
the reconciler can see a difference in specs and applies the parallelism
overrides. I can submit a PR if assigned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
