rlbabyuk opened a new issue, #36218:
URL: https://github.com/apache/airflow/issues/36218
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.5.3
### What happened?
While running dag with do_xcom_push=True parameter, after pod has been
created, airflow task marked as failed, but the pod continues running, so the
base container is completed and the process is stuck on the xcom_sidecar
container.
I've checked and the sidecar airflow container has file
/airflow/xcom/return.json with content {"exit_code": "0", "errors": "",
"successful_target_ids": []}
No error in the logs
In some cases task stops pod tracking while pod get created, in other case
on xcom_sidecar already running
<img width="903" alt="Screenshot 2023-12-14 at 11 09 40"
src="https://github.com/apache/airflow/assets/26601785/adca5886-9398-46e5-9045-f477815172e1";>
<img width="891" alt="Screenshot 2023-12-14 at 11 08 18"
src="https://github.com/apache/airflow/assets/26601785/4bff546e-51cf-4264-84b4-3c792ef8c2dc";>
Also in my case dag has
container_resources=k8s.V1ResourceRequirements(limits={"memory": "3G"})
Tried to increase the limit but it didn't help
### What you think should happen instead?
Task should not fail and pod should be deleted
### How to reproduce
Create kubernetes_pod_operator dag with the following parameters:
do_xcom_push=True,
on_finish_action="delete_pod"
Start the dag simultaneously in 10 threads
Check the container statuses and airflow logs for failed runs
### Operating System
GKE Container-Optimized OS
### Versions of Apache Airflow Providers
"apache-airflow==2.5.3",
"apache-airflow-providers-common-sql==1.8.0",
"apache-airflow-providers-cncf-kubernetes==7.3.0",
"apache-airflow-providers-ftp==3.6.0",
"apache-airflow-providers-google==10.10.0",
"apache-airflow-providers-http==4.6.0",
"apache-airflow-providers-imap==3.4.0",
"apache-airflow-providers-sqlite==3.5.0",
### Deployment
Google Cloud Composer
### Deployment details
Used image -
composer-2.5.0-airflow - 2.5.3
### Anything else?
Reproduces randomly and often
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
