[
https://issues.apache.org/jira/browse/FLINK-40587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyula Fora updated FLINK-40587:
-------------------------------
Fix Version/s: kubernetes-operator-1.16.1
> Recreating a FlinkDeployment throws an autoscaler exception
> -----------------------------------------------------------
>
> Key: FLINK-40587
> URL: https://issues.apache.org/jira/browse/FLINK-40587
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Assignee: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.17.0, kubernetes-operator-1.16.1
>
>
> Currently, deleting and creating a FlinkDeloyment that has a previous
> autoscaler state (aka autoscaler ConfigMap owned by the resource is in place)
> expose the following behavior:
> - Throws the following exception:
> {code:bash}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https://10.96.0.1:443/api/v1/namespaces/<namespace>/configmaps/autoscaler-<flinkdeployment-name>.
> Message: configmaps "autoscaler-<flinkdeployment-name>" not found. Received
> status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[],
> group=null, kind=configmaps, name=autoscaler-<flinkdeployment-name>,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=configmaps "autoscaler-<flinkdeployment-name>" not found,
> metadata=ListMeta(_continue=null, remainingItemCount=null,
> resourceVersion=null, selfLink=null, shardInfo=null,
> additionalProperties={}), reason=NotFound, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda<span>handleResponse</span>0(OperationSupport.java:549)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniApply.tryFire(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
> at
> io.fabric8.kubernetes.client.http.StandardHttpClient.lambda</span>completeOrCancel<span>10(StandardHttpClient.java:149)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
> at
> io.fabric8.kubernetes.client.utils.AsyncUtils.lambda<span>retryWithExponentialBackoff</span>3(AsyncUtils.java:91)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture<span>Completion.exec(Unknown
> Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
> at
> java.base/java.util.concurrent.ForkJoinPool</span>WorkQueue.topLevelExec(Unknown
> Source)
> at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
> at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
> {code}
> - Emit the the following event/log:
> {code:java}
> INFO org.apache.flink.kubernetes.operator.listener.AuditUtils - >>>
> Event[Job] | Warning | AUTOSCALERERROR | Failure executing: ...
> {code}
> The root cause comes from the fact that the operator autoscaler mechanism
> keeps the autoscaler ConfigMap cached, and doesn't react when the Kubernetes
> garbage-collects the FlinkDeployment-owned resources. The cache is keyed by
> namespace and name, not by owner UUID, and it is only cleared on the error
> path, so after a recreate the store keeps writing through a view of a
> ConfigMap that no longer exists, receiving a 404 REST response.
> On top of that, a 404 REST response on flush is treated like any other
> failure. As it is expected here, it does not make sense to treat it the same
> as the other exceptions and mark it as an AUTOSCALERERROR event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
