alaiou commented on issue #11841: URL: https://github.com/apache/airflow/issues/11841#issuecomment-716701907
We also encountered this issue. Turns out the root cause was the newest release of the k8s python client https://github.com/kubernetes-client/python/releases/tag/v12.0.0. Code was added to handle the 410 status code and raise an Exception in this [PR](https://github.com/kubernetes-client/python-base/pull/133). In Airflow however, the `KubernetesJobWatcher` is expecting an event, which is would then handle the status code gracefully, by resetting the resource_version number in `process_error`. It never actually gets to that point in the code. As you can see the Exception thrown is from [here](https://github.com/kubernetes-client/python-base/blob/2da2b981ca806b25487ad92d01a2164815c18517/watch/watch.py#L176). Our work around was to explicitly use the previous version of the k8s python client, by using the appropriate constrained/"know-to-be-working" version of Airflow and its libraries. ``` pip install \ apache-airflow[kubernetes]==1.10.12 \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"; ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
