tikkanipraveen opened a new issue, #29237:
URL: https://github.com/apache/airflow/issues/29237
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
All our airflow tasks which takes more than 5 mins are getting killed with
SIGTERM error. Tasks which take less than 5 mins are getting successfully
executed.
We tried to set up new airflow instances on a new AKS namespace as well and
same issue is appearing.
Current version: 2.3.2
We tried this on 2.3.4 and there also we have same issue.
Error:
[2023-01-30, 03:01:22 UTC] {kubernetes_pod.py:368} INFO - creating pod with
labels {'dag_id': 'XXX_load_batch_XXX_XXX', 'task_id': 'jobcluster_xxx_xxx',
'execution_date':
'2023-01-30T0200000000-6d45593a7', 'try_number': '1'} and launcher
<airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at
0x7fea17527550>
[2023-01-30, 03:01:22 UTC] {pod_launcher.py:198} INFO - Event:
jobcluster-execution-task-inve-e28a82c1e7944dec8b9deb6807c7b3fb had an event of
type Pending
[2023-01-30, 03:01:22 UTC] {pod_launcher.py:128} WARNING - Pod not yet
started: jobcluster-execution-task-inve-e28a82c1e7944dec8b9deb6807c7b3fb
[2023-01-30, 03:01:23 UTC] {pod_launcher.py:198} INFO - Event:
jobcluster-execution-task-inve-e28a82c1e7944dec8b9deb6807c7b3fb had an event of
type Running
[2023-01-30, 03:06:07 UTC] {local_task_job.py:84} ERROR - Received SIGTERM.
Terminating subprocesses
[2023-01-30, 03:06:07 UTC] {process_utils.py:125} INFO - Sending
Signals.SIGTERM to group 29. PIDs of all processes in the group: [29]
[2023-01-30, 03:06:07 UTC] {process_utils.py:80} INFO - Sending the signal
Signals.SIGTERM to group 29
[2023-01-30, 03:06:07 UTC] {taskinstance.py:1553} ERROR - Received SIGTERM.
Terminating subprocesses.
[2023-01-30, 03:06:07 UTC] {taskinstance.py:1902} ERROR - Task failed with
exception
We have tried to modify airflow config parameters as mentioned in other
posts like below but still no luck.
https://github.com/apache/airflow/issues/14672
https://github.com/apache/airflow/issues/18041
https://stackoverflow.com/questions/73254161/kubernetes-executor-task-is-receiving-sigterm-signal-if-it-runs-more-than-5-mi/73255132#73255132
### What you think should happen instead
All Tasks should execute successfully without any SIGTERM errors.
### How to reproduce
_No response_
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployment is on Azure Kubernetes (AKS)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
