njandu opened a new issue, #30198:
URL: https://github.com/apache/airflow/issues/30198
### Apache Airflow version
2.5.2
### What happened
In the newer versions of airflow, When airflow scheduler is restarted via
daemon process. The tasks currently running do not get marked as failed.
Instead they re-queue. In cases such as an SSHoperator being used. The task
restarts and resends the command. Without the previous task marked as failed.
Using LocalExecutor with postgres db.
Command to restart
`sudo systemctl restart airflow-scheduler`
Below is the error output. It no longer marks the task as failed. Instead it
clears the task and starts again.
This is the error output----------------------------------------------
[2023-03-16, 13:21:26 UTC] {transport.py:1873} INFO - Authentication
(publickey) successful!
[2023-03-16, 13:21:26 UTC] {ssh.py:476} INFO - Running command: sleep 1000
[2023-03-16, 13:21:30 UTC] {taskinstance.py:1482} ERROR - Received SIGTERM.
Terminating subprocesses. # Occurs when I restart
[2023-03-16, 13:21:30 UTC] {logging_mixin.py:137} INFO - no action
[2023-03-16, 13:21:30 UTC] {local_task_job.py:113} ERROR - Received SIGTERM.
Terminating subprocesses
[2023-03-16, 13:21:30 UTC] {process_utils.py:130} INFO - Sending
Signals.SIGTERM to group 356481. PIDs of all processes in the group: [356481]
[2023-03-16, 13:21:30 UTC] {process_utils.py:84} INFO - Sending the signal
Signals.SIGTERM to group 356481
[2023-03-16, 13:21:30 UTC] {taskinstance.py:1482} ERROR - Received SIGTERM.
Terminating subprocesses.
[2023-03-16, 13:21:30 UTC] {standard_task_runner.py:100} ERROR - Failed to
execute job 122 for task test_task (Task received SIGTERM signal; 356481)
[2023-03-16, 13:21:30 UTC] {process_utils.py:79} INFO - Process
psutil.Process(pid=356481, status='terminated', exitcode=1, started='13:21:25')
(356481) terminated with exit code 1
[2023-03-16, 13:21:30 UTC] {local_task_job.py:212} INFO - Task exited with
return code 143
[2023-03-16, 13:21:30 UTC] {taskinstance.py:2585} INFO - 0 downstream tasks
scheduled from follow-on schedule check
The task two then starts up automatically -------------------------
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1084} INFO - Dependencies all
met for dep_context=non-requeueable deps ti=<TaskInstance: test_dag.test_task
manual__2023-03-16T13:21:21.231337+00:00 [queued]>
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1084} INFO - Dependencies all
met for dep_context=requeueable deps ti=<TaskInstance: test_dag.test_task
manual__2023-03-16T13:21:21.231337+00:00 [queued]>
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1282} INFO -
--------------------------------------------------------------------------------
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1283} INFO - Starting attempt 2
of 1
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1284} INFO -
--------------------------------------------------------------------------------
[2023-03-16, 13:21:34 UTC] {taskinstance.py:1303} INFO - Executing
<Task(SSHOperator): test_task> on 2023-03-16 13:21:21.231337+00:00
[2023-03-16, 13:21:34 UTC] {standard_task_runner.py:55} INFO - Started
process 357161 to run task
[2023-03-16, 13:21:34 UTC] {standard_task_runner.py:82} INFO - Running:
['airflow', 'tasks', 'run', 'test_dag', 'test_task',
'manual__2023-03-16T13:21:21.231337+00:00', '--job-id', '124', '--raw',
'--subdir', 'DAGS_FOLDER/test_dag.py', '--cfg-path', '/tmp/tmpii4ulvqs']
[2023-03-16, 13:21:34 UTC] {standard_task_runner.py:83} INFO - Job 124:
Subtask test_task
[2023-03-16, 13:21:34 UTC] {task_command.py:388} INFO - Running
<TaskInstance: test_dag.test_task manual__2023-03-16T13:21:21.231337+00:00
[running]> on host 172.31.18.39
[2023-03-16, 13:21:35 UTC] {taskinstance.py:1509} INFO - Exporting the
following env vars:
### What you think should happen instead
When airflow is scheduler service is restarted tasks should fail. There
should be a manual intervention to mark as success or clear the task.
In bold is what should happen to the task
----------------------------
[2023-03-20, 13:17:22 UTC] {process_utils.py:84} INFO - Sending the signal
Signals.SIGTERM to group 418442
**[2023-03-20, 13:17:22 UTC] {taskinstance.py:1321} INFO - Marking task as
FAILED. dag_id=test_dag, task_id=test_task, execution_date=20230320T131707,
start_date=20230320T131708, end_date=20230320T131722**
[2023-03-20, 13:17:22 UTC] {standard_task_runner.py:100} ERROR - Failed to
execute job 199 for task test_task (Task received SIGTERM signal; 418442)
### How to reproduce
Airflow v2.5 or above
local executer
ssh conn setup
venv setup
DAG:
from airflow import DAG
from airflow.contrib.operators.ssh_operator import SSHOperator
from datetime import datetime, time, timedelta
dag = DAG(
dag_id="test_dag",
schedule_interval="00 12 * * 7",
catchup=False,
# dagrun_timeout=timedelta(minutes=5),
description="test restart",
)
DownloadFactsetFiles = SSHOperator(
ssh_conn_id="NJ_con",
task_id="test_task",
cmd_timeout=1000,
execution_timeout=timedelta(hours=23, minutes=55),
command=" sleep 1000 ",
dag=dag,
)
### Operating System
NAME="Ubuntu" VERSION="20.04.2 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS" VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"; SUPPORT_URL="https://help.ubuntu.com/";
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
VERSION_CODENAME=focal UBUNTU_CODENAME=focal
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==2.6.0
apache-airflow-providers-common-sql==1.3.4
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-http==4.2.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-sqlite==3.3.1
apache-airflow-providers-ssh==3.5.0
### Deployment
Virtualenv installation
### Deployment details
Virtualenvs
### Anything else
Occurs every restart
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
