jkinkead opened a new issue #11841:
URL: https://github.com/apache/airflow/issues/11841
**Apache Airflow version**: 1.10.11
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```
Server Version: version.Info{Major:"1", Minor:"15+",
GitVersion:"v1.15.11-eks-065dce",
GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean",
BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc",
Platform:"linux/amd64"}
```
**What happened**:
We've been seeing occasional issues in our logs where the Kubernetes
executor throws an API exception [on this stream
call](https://github.com/apache/airflow/blob/2d25377a20cabac9ea1d26d015e6ac6e9606ac14/airflow/executors/kubernetes_executor.py#L298-L299):
```
[2020-10-25 15:59:15,636] {{kubernetes_executor.py:277}} ERROR - Unknown
error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py",
line 271, in run
self.worker_uuid, self.kube_config)
File
"/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py",
line 299, in _run
**kwargs):
File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py",
line 177, in stream
status=obj['code'], reason=reason)
kubernetes.client.exceptions.ApiException: (410)
Reason: Gone: too old resource version: 46672510 (46702381)
```
This is a normal response (and [handled in the `process_error`
method](https://github.com/apache/airflow/blob/2d25377a20cabac9ea1d26d015e6ac6e9606ac14/airflow/executors/kubernetes_executor.py#L326-L332)),
and should be handled gracefully, probably like the event is (catching &
resetting `self.resource_version`).
**Anything else we need to know**:
This seems to be triggered by having very long-running (multiple days old)
task pods in our system. These aren't normal operations, but were the result of
some deadlocking bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
