Hi,
I'm occasionally trying to force the final task in a DAG by clicking on the
task in the web UI and clicking 'Run' (with 'Force' and 'Ignore
Dependencies' selected). The UI shows 'Sent <TaskInstance: ...> to the
message queue, it should start any moment now'. Going back to that task
and selecting 'Log' I see something like this:
[2016-06-15 20:11:44,172] {models.py:154} INFO - Filling up the DagBag
from /home/airflow/workspace/verticadw/airflow/dags/etl_gsn_daily_kpi_email.py
[2016-06-15 20:11:44,881] {base_hook.py:53} INFO - Using connection
to: localhost
No sign of an error but no sign of success. But if I look at the Celery
Flower dashboard, I can see this resulted in a failure with result:
AirflowException('Celery command failed',)
If I 'Clear'|'Downstream' the task, the task succeeds. And the task run in
the scheduled DAG succeeds.
Example args for failed tasks:
[u'airflow run etl_gsn_daily_kpi_email t_final_send_email
2016-06-14T10:00:00 --force --local -sd
DAGS_FOLDER/etl_gsn_daily_kpi_email.py ']
[u'airflow run etl_gsn_daily_kpi_email t_final_send_email
2016-06-14T10:00:00 -i --force --local -sd
DAGS_FOLDER/etl_gsn_daily_kpi_email.py ']
Example args for successful tasks:
[u'airflow run etl_gsn_daily_kpi_email t_final_send_email
2016-06-14T10:00:00 --local -sd DAGS_FOLDER/etl_gsn_daily_kpi_email.py ']
and the log:
[2016-06-15 19:44:46,147] {models.py:1219} INFO - Executing
<Task(PythonOperator): t_final_send_email> on 2016-06-14 10:00:00
[2016-06-15 19:44:51,979] {python_operator.py:67} INFO - Done.
Returned value was: [Airflow] GSN Games Metrics 2016-06-14 sent to
('dobr...@gsngames.com',)
I looked in the logs (airflow-flower.log, airflow-scheduler.log,
airflow-webserver.log, and airflow-worker.log) but didn't find any clues.
I'm running airflow v1.7.1.3, using CeleryExecutor backed by redis.
Celery portion of airflow.cfg:
[celery]
celery_app_name = airflow.executors.celery_executor
celeryd_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://localhost
celery_result_backend = redis://localhost
flower_port = 5555
default_queue = default
Any advice on debugging this?
thanks,
Dennis
