mbig opened a new issue, #25745:
URL: https://github.com/apache/airflow/issues/25745
### Apache Airflow version
Other Airflow 2 version
### What happened
We are upgrading from airflow 1.10.15 to 2.1.4 and we getting this error on
all our dags after they successfully finish.
We are also getting "Error when executing on_failure_callback" and "Error
when executing on_retry_callback" when a task fails and wants to retry.
[2022-08-16 15:35:08,370] {taskinstance.py:1350} ERROR - Error when
executing on_success_callback
Traceback (most recent call last):
File
"/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line
1348, in _run_finished_callback
task.on_success_callback(context)
File "/usr/local/airflow/dagbag/repo/dags/pod_utility.py", line 110, in
on_success_callback
terminate_pod_from_context(context=context, kube_context=kube_context)
File "/usr/local/airflow/dagbag/repo/dags/pod_utility.py", line 227, in
terminate_pod_from_context
container_statuses = get_container_statuses_from_pod(context)
File "/usr/local/airflow/dagbag/repo/dags/pod_utility.py", line 77, in
get_container_statuses_from_pod
pod_data = read_pod(
File "/usr/local/airflow/dagbag/repo/dags/pod_utility.py", line 340, in
read_pod
return client.read_namespaced_pod(name=name, namespace=namespace)
File
"/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py",
line 19078, in read_namespaced_pod
(data) = self.read_namespaced_pod_with_http_info(name, namespace,
**kwargs) # noqa: E501
File
"/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py",
line 19155, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File
"/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line
340, in call_api
return self.__call_api(resource_path, method,
File
"/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line
172, in __call_api
response_data = self.request(
File
"/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line
362, in request
return self.rest_client.GET(url,
File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py",
line 237, in GET
return self.request("GET", url,
File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py",
line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
### What you think should happen instead
Also the task is marked as being successfully, we should not get the failure
in the logs or if the task fails, it should try to rerun.
### How to reproduce
Install airflow 2.1.4
Run test dags
### Operating System
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux"
VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian
HOME_URL="https://www.debian.org/"; SUPPORT_URL="https://www.debian.org/support";
BUG_REPORT_URL="https://bugs.debian.org/";
### Versions of Apache Airflow Providers
apache-airflow[crypto,postgres,ssh,kubernetes,redis,oauth,datadog,statsd,s3,amazon]==2.1.4
apache-airflow-providers-cncf-kubernetes==2.2.0
apache-airflow-providers-celery==2.0.0
celery==4.4.7
authlib>=0.15.3
ddtrace==0.27.0
dumb-init==1.2.2
flask-appbuilder>=2.3.4
markupsafe==1.1.1
more-itertools==7.2.0
ndg-httpsclient==0.5.1
ordereddict==1.1
pre-commit==2.6.0
psycopg2>=2.8.3
pypd==1.1.0
pyOpenSSL==19.0.0
# pin this lower than 1.4 to avoid problems with sqlAlchemy utilities.
# apache-airflow has sqlAlchemy as a dependency
SQLAlchemy==1.3.23
datadog<=0.42.0
WTForms<3
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
