tanvn commented on issue #39088:
URL: https://github.com/apache/airflow/issues/39088#issuecomment-2063070213
I managed to reproduce this issue on my development environment by creating
a DAG with about 100 tasks. When there are about 15 running tasks, I run a helm
command to upgrade the deployment, then the old scheduler is terminated and a
new one is created and running.
The log shows that
```
[2024-04-18T05:45:33.367+0000] {kubernetes_executor.py:749} INFO -
attempting to adopt pod orphaned-test-ba20047562f24ec993a3816b7d17cfd7
...
[2024-04-18T05:45:33.570+0000] {kubernetes_executor.py:220} INFO - Event:
orphaned-test-ba20047562f24ec993a3816b7d17cfd7 is Running
```
but later, the corresponding task is still reset:
```
[2024-04-18T05:45:38.658+0000] {scheduler_job.py:1463} INFO - Reset the
following 8 orphaned TaskInstances:
....
```
And the task is terminated later
```
[2024-04-18, 14:45:58 JST] {local_task_job.py:223} WARNING - State of this
instance has been externally set to queued. Terminating instance.
[2024-04-18, 14:45:58 JST] {process_utils.py:129} INFO - Sending
Signals.SIGTERM to group 34. PIDs of all processes in the group: [34]
[2024-04-18, 14:45:58 JST] {process_utils.py:84} INFO - Sending the signal
Signals.SIGTERM to group 34
[2024-04-18, 14:45:58 JST] {taskinstance.py:1483} ERROR - Received SIGTERM.
Terminating subprocesses.
[2024-04-18, 14:45:58 JST] {taskinstance.py:1772} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/plugins/operators/my_dummy_operator.py", line 95, in
execute
time.sleep(300)
File
"/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line
1485, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
```
The cause is remaining unclear but I can confirm that the issue can be
reproduced (so maybe this is a bug).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
