andormarkus commented on issue #13824:
URL: https://github.com/apache/airflow/issues/13824#issuecomment-883239510
Hi @ashb
I set `AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION=False` in Airflow
2.1.2 which does not have #16289.
All the tasks are successful on Airflow side, however in flower all are
failed.
The logs are showing up in CloudWatch as expected.
My airflow.cfg
```bash
[scheduler]
dag_dir_list_interval = 60
parsing_processes = 4
run_duration = 41460
schedule_after_task_execution = False
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow
```
This is how Celery logs look like for a given worker:
```bash
[2021-07-20 09:02:49,057: INFO/ForkPoolWorker-15] Executing command in
Celery: ['airflow', 'tasks', 'run', 'XXXXXXXX', 'eks.sensor',
'2021-07-20T08:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir',
'/opt/airflow/dags/XXXXXXXX.py']
[2021-07-20 09:02:49,086: INFO/ForkPoolWorker-15] Filling up the DagBag from
/opt/airflow/dags/XXXXXXXX.py
[2021-07-20 09:02:51,442: INFO/ForkPoolWorker-15] Datasets List: 2
[2021-07-20 09:02:51,442: INFO/ForkPoolWorker-15] Start getting tables list
from dataset: XXXXXXXX
[2021-07-20 09:02:51,585: INFO/ForkPoolWorker-15] Start getting tables list
from dataset: XXXXXXXX
[2021-07-20 09:02:52,380: WARNING/ForkPoolWorker-15] Running <TaskInstance:
XXXXXXXX.eks.sensor 2021-07-20T08:00:00+00:00 [queued]> on host
airflow-worker-7c6b4f75f9-x67r9
[2021-07-20 09:03:18,710: ERROR/ForkPoolWorker-15] Failed to execute task
Task received SIGTERM signal.
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py",
line 117, in _execute_in_fork
args.func(args)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line
91, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 238, in task_run
_run_task_by_selected_method(args, dag, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 121, in _run_task_by_local_task_job
run_job.run()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py",
line 245, in run
self._execute()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py",
line 100, in _execute
self.task_runner.start()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/task/task_runner/standard_task_runner.py",
line 41, in start
self.process = self._start_by_fork()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/task/task_runner/standard_task_runner.py",
line 92, in _start_by_fork
logging.shutdown()
File "/usr/local/lib/python3.8/logging/__init__.py", line 2126, in shutdown
h.flush()
File
"/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line
297, in flush
q.join()
File "/usr/local/lib/python3.8/queue.py", line 89, in join
self.all_tasks_done.wait()
File "/usr/local/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1286, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-07-20 09:04:18,738: ERROR/ForkPoolWorker-15] Failed to execute task
[Errno 2] No such file or directory: '/tmp/tmpbwn0h8za'.
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py",
line 117, in _execute_in_fork
args.func(args)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line
91, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 238, in task_run
_run_task_by_selected_method(args, dag, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 121, in _run_task_by_local_task_job
run_job.run()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py",
line 245, in run
self._execute()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py",
line 145, in _execute
self.on_kill()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py",
line 166, in on_kill
self.task_runner.on_finish()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/task/task_runner/base_task_runner.py",
line 178, in on_finish
self._error_file.close()
File "/usr/local/lib/python3.8/tempfile.py", line 499, in close
self._closer.close()
File "/usr/local/lib/python3.8/tempfile.py", line 436, in close
unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpbwn0h8za'
[2021-07-20 09:04:18,834: ERROR/ForkPoolWorker-15] Task
airflow.executors.celery_executor.execute_command[0a8ca64c-01df-4868-a21d-b369d3f7a6cd]
raised unexpected: AirflowException('Celery command failed on host:
airflow-worker-7c6b4f75f9-x67r9')
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/celery/app/trace.py", line
412, in trace_task
R = retval = fun(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/celery/app/trace.py", line
704, in __protected_call__
return self.run(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py",
line 88, in execute_command
_execute_in_fork(command_to_exec)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py",
line 99, in _execute_in_fork
raise AirflowException('Celery command failed on host: ' +
get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host:
airflow-worker-7c6b4f75f9-x67r9
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
