Alex Kukhar created FLINK-41222:
-----------------------------------
Summary: Autoscaler reports an error when a ConfigMap write is
applied but the client's retry conflicts
Key: FLINK-41222
URL: https://issues.apache.org/jira/browse/FLINK-41222
Project: Flink
Issue Type: Bug
Components: Autoscaler, Kubernetes Operator
Affects Versions: kubernetes-operator-1.16.1, kubernetes-operator-1.15.0
Environment: Flink Kubernetes Operator 1.15.0 (fabric8
kubernetes-client 7.3.1), GKE control plane 1.35.6-gke.1250001
Reporter: Alex Kukhar
The autoscaler stores its state in a ConfigMap through
{{{}ConfigMapView.flush(){}}}. If the API server applies a write but its
response arrives after the client's request timeout (fabric8 default 10 s), the
fabric8 client retries the request. The retry fails with 409. An update fails
because its {{resourceVersion}} is now stale, and a create fails because the
ConfigMap already exists. {{ConfigMapStore.flush()}} then drops its cache entry
and rethrows. {{JobAutoScalerImpl}} counts an autoscaler error and emits an
{{AutoscalerError}} event, even though Kubernetes already holds the state the
operator wrote.
We saw this on GKE while control-plane storage writes stalled for 10–25 s. The
retried autoscaler ConfigMap updates failed with 409 "the object has been
modified", and no other writer had changed those ConfigMaps in between.
Proposed fix: when the update or the create gets a 409, re-read the ConfigMap
and adopt it if its data and owner references equal the local state. Any other
conflict is rethrown as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
