Dennis-Mircea Ciupitu created FLINK-40587:
---------------------------------------------
Summary: Recreating a FlinkDeployment throws an autoscaler
exception
Key: FLINK-40587
URL: https://issues.apache.org/jira/browse/FLINK-40587
Project: Flink
Issue Type: Improvement
Components: Autoscaler, Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
Currently, deleting and creating a FlinkDeloyment that has a previous
autoscaler state (aka autoscaler ConfigMap owned by the resource is in place)
expose the following behavior:
- Throws the following exception:
{code:bash}
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
executing: PUT at:
https://10.96.0.1:443/api/v1/namespaces/str-local/configmaps/autoscaler-<flinkdeployment-name>.
Message: configmaps "autoscaler-<flinkdeployment-name>" not found. Received
status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[],
group=null, kind=configmaps, name=autoscaler-<flinkdeployment-name>,
retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
message=configmaps "autoscaler-<flinkdeployment-name>" not found,
metadata=ListMeta(_continue=null, remainingItemCount=null,
resourceVersion=null, selfLink=null, shardInfo=null, additionalProperties={}),
reason=NotFound, status=Failure, additionalProperties={}).
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda<span>handleResponse</span>0(OperationSupport.java:549)
at
java.base/java.util.concurrent.CompletableFuture<span>UniApply.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) at
io.fabric8.kubernetes.client.http.StandardHttpClient.lambda</span>completeOrCancel<span>10(StandardHttpClient.java:149)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
at
io.fabric8.kubernetes.client.utils.AsyncUtils.lambda<span>retryWithExponentialBackoff</span>3(AsyncUtils.java:91)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>Completion.exec(Unknown
Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at
java.base/java.util.concurrent.ForkJoinPool</span>WorkQueue.topLevelExec(Unknown
Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
{code}
- Emit the the following event/log:
{code}
INFO org.apache.flink.kubernetes.operator.listener.AuditUtils - >>> Event[Job]
| Warning | AUTOSCALERERROR | Failure executing: ...
{code}
The root cause comes from the fact that the operator autoscaler mechanism keeps
the autoscaler ConfigMap cached, and doesn't react when the Kubernetes
garbage-collects the FlinkDeployment-owned resources. The cache is keyed by
namespace and name, not by owner UUID, and it is only cleared on the error
path, so after a recreate the store keeps writing through a view of a ConfigMap
that no longer exists, receiving a 404 REST response.
On top of that, a 404 REST response on flush is treated like any other failure.
As it is expected here, it does not make sense to treat it the same as the
other exceptions and mark it as an AUTOSCALERERROR event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
