Hi,
We use airflow 1.7.1.3 with Celery (postgress as its backend DB).
After a few hours, we noticed that no task is executed.
Some tasks failed before it happened.
from worker log:
[2016-08-28 11:31:33,300] {__init__.py:36} INFO - Using executor
CeleryExecutor
Logging into:
/var/log/airflow//daily_agg/activities_per_day_in_week_task/2016-08-19T02:00:00
[2016-08-28 11:31:34,663] {__init__.py:36} INFO - Using executor
CeleryExecutor
Traceback (most recent call last):
File "/usr/bin/airflow", line 15, in <module>
args.func(args)
File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 237, in
run
pool=args.pool,
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in
wrapper
result = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1245, in
run
result = task_copy.execute(context=context)
File
"/usr/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line
83, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
When we tried to see airflow processes, we saw that there are processes
related to postgres in state of 'idle'
[root@hadoop01 ~]# ps -ef | grep airflow
root 12884 3772 14 08:11 pts/7 00:30:40 /usr/bin/python
/usr/bin/airflow scheduler
root 12918 12901 0 08:11 pts/8 00:00:04 /usr/bin/python
/usr/bin/airflow serve_logs
root 13577 13253 0 08:11 pts/10 00:00:05 gunicorn: master
[airflow-webserver]
root 13943 13577 0 08:11 pts/10 00:00:04 gunicorn: worker
[airflow-webserver]
root 13945 13577 0 08:11 pts/10 00:00:07 gunicorn: worker
[airflow-webserver]
root 13950 13577 0 08:11 pts/10 00:00:06 gunicorn: worker
[airflow-webserver]
root 13954 13577 0 08:11 pts/10 00:00:05 gunicorn: worker
[airflow-webserver]
postgres 22139 1847 0 11:39 ? 00:00:00 postgres: airflow
airflow_db ::1(52324) idle
postgres 22312 1847 0 11:39 ? 00:00:00 postgres: airflow
airflow_db ::1(52365) idle
postgres 24844 1847 0 11:47 ? 00:00:00 postgres: airflow
airflow_db ::1(53002) idle in transaction
root 24849 12222 0 11:47 pts/9 00:00:00 grep --color=auto airflow
postgres 55388 1847 0 11:10 ? 00:00:00 postgres: airflow
airflow_db ::1(47759) idle
postgres 61412 1847 0 11:13 ? 00:00:00 postgres: airflow
airflow_db ::1(48266) idle
any idea?
Thanks,
Hila
