mayuribh3 commented on issue #14672:
URL: https://github.com/apache/airflow/issues/14672#issuecomment-1122701086
Facing similar issue in 2.3.0
[2022-05-06 12:57:56,230] {spark_kubernetes.py:121} INFO - Spark application
is still in state: RUNNING
[2022-05-06 12:58:02,199] {local_task_job.py:82} ERROR - Received SIGTERM.
Terminating subprocesses
[2022-05-06 12:58:02,269] {process_utils.py:100} INFO - Sending
Signals.SIGTERM to GPID 859
[2022-05-06 12:58:02,269] {taskinstance.py:1411} ERROR - Received SIGTERM.
Terminating subprocesses.
[2022-05-06 12:58:02,491] {taskinstance.py:1703} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1514, in _execute_task
result = execute_callable(context=context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/base.py",
line 247, in execute
time.sleep(self._get_next_poke_interval(started_at, run_duration,
try_number))
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1413, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-05-06 12:58:02,578] {taskinstance.py:1280} INFO - Marking task as
FAILED. dag_id=bda-plab1-bda-azr-dev-wu2-job-sql-collector-samson-csmapp7,
task_id=spark_pi_monitor, execution_date=20220506T125428,
start_date=20220506T125453, end_date=20220506T125802
[2022-05-06 12:58:02,774] {standard_task_runner.py:91} ERROR - Failed to
execute job 65583 for task spark_pi_monitor
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py",
line 85, in _start_by_fork
args.func(args, dag=self.dag)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line
92, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 184, in _run_raw_task
error_file=args.error_file,
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 70, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1514, in _execute_task
result = execute_callable(context=context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/base.py",
line 247, in execute
time.sleep(self._get_next_poke_interval(started_at, run_duration,
try_number))
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1413, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-05-06 12:58:02,807] {process_utils.py:66} INFO - Process
psutil.Process(pid=859, status='terminated', exitcode=1, started='12:54:53')
(859) terminated with exit code 1
[2022-05-06 12:58:02,807] {local_task_job.py:154} INFO - Task exited with
return code 143
[2022-05-06 12:58:03,139] {local_task_job.py:264} INFO - 0 downstream tasks
scheduled from follow-on schedule check
Already tried below parameters but still issue persist:
AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME: "604800"
AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False"
AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC
scheduler_heartbeat_sec
scheduler_health_check_threshold
Please suggest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
