[GitHub] [airflow] andormarkus commented on issue #13824: Cloudwatch Integration: SIGTERM/SIGKILL Sent Following DAG Completion, Causing Errors in Worker Logs

Tue, 01 Mar 2022 11:27:46 -0800 


andormarkus commented on issue #13824:
URL: https://github.com/apache/airflow/issues/13824#issuecomment-1055782143


   Hi @rafidka, thanks, for the update. 
   
   Here is not my current findings:
   
   Used versions:
   ```yaml
   airflow: 2.2.4
   watchtower: 2.0.1
   ```
   
   Test setup: I'm running 5 [simple 
dag](https://github.com/apache/airflow/issues/13824#issuecomment-1012838195) 
every in the past few days:
   
   <img width="1731" alt="Screen Shot 2022-03-01 at 19 31 50" 
src="https://user-images.githubusercontent.com/51825189/156227973-6f3bba96-9581-4b9c-bda4-07d389c97550.png";>
   
   From flower perspective everything looks good:
   <img width="1727" alt="Screen Shot 2022-03-01 at 19 33 12" 
src="https://user-images.githubusercontent.com/51825189/156228089-070f61cf-03d4-4cbe-b7c4-61cdecedd66e.png";>
   
   
   I have checked the worker logs and few times per hour I get the following 
error messages:
   ```bash
   [2022-03-01 18:21:24,954: ERROR/ForkPoolWorker-15] Failed to execute task 
Task received SIGTERM signal.
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/celery_executor.py",
 line 121, in _execute_in_fork
       args.func(args)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", 
line 48, in command
       return func(*args, **kwargs)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 
92, in wrapper
       return f(*args, **kwargs)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py",
 line 298, in task_run
       _run_task_by_selected_method(args, dag, ti)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py",
 line 105, in _run_task_by_selected_method
       _run_task_by_local_task_job(args, ti)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py",
 line 163, in _run_task_by_local_task_job
       run_job.run()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/base_job.py", 
line 246, in run
       self._execute()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/local_task_job.py",
 line 103, in _execute
       self.task_runner.start()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/task/task_runner/standard_task_runner.py",
 line 41, in start
       self.process = self._start_by_fork()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/task/task_runner/standard_task_runner.py",
 line 97, in _start_by_fork
       logging.shutdown()
     File "/usr/local/lib/python3.9/logging/__init__.py", line 2141, in shutdown
       h.flush()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/watchtower/__init__.py", line 
432, in flush
       q.join()
     File "/usr/local/lib/python3.9/queue.py", line 90, in join
       self.all_tasks_done.wait()
     File "/usr/local/lib/python3.9/threading.py", line 312, in wait
       waiter.acquire()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py",
 line 1415, in signal_handler
   ```
   
   
   Worker logs for a failed task looks like this:
   ```bash
   ▶ kubectl -n airflow logs airflow-worker worker | grep ForkPoolWorker-16
   
   [2022-03-01 19:08:00,654: INFO/ForkPoolWorker-16] Celery task ID: 
fee17d13-2423-4ed1-ab2f-3f1a3fd34551
   [2022-03-01 19:08:00,709: INFO/ForkPoolWorker-16] Filling up the DagBag from 
/opt/airflow/dags/repo/dags/simple_dag_2.py
   [2022-03-01 19:08:00,842: WARNING/ForkPoolWorker-16] Running <TaskInstance: 
simple_dag_2.sleep scheduled__2022-03-01T19:07:00+00:00 [queued]> on host 
airflow-worker-58b8d8789b-w7jwv
   [2022-03-01 19:08:24,702: ERROR/ForkPoolWorker-16] Failed to execute task 
Task received SIGTERM signal.
   [2022-03-01 19:08:25,170: INFO/ForkPoolWorker-16] Task 
airflow.executors.celery_executor.execute_command[fee17d13-2423-4ed1-ab2f-3f1a3fd34551]
 succeeded in 24.535810169007163s: None
   ```
   
   @rafidka Where should I find the `State of this instance has been externally 
set to success. Terminating instance.` warning Message? I can not find in the 
worker or scheduler logs.
   
   Based on my testing `Celery command failed on host` error was fixed with 
Airflow 2.2.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to