[
https://issues.apache.org/jira/browse/AIRFLOW-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277538#comment-15277538
]
Jeremiah Lowin commented on AIRFLOW-84:
---------------------------------------
Usually if the Celery command failed it means Airflow couldn't even start the
task, or crashed during task execution. As you suggest, I've seen that happen
with out of memory errors causing the celery workers to fail before running the
task.
> Queued tasks in Airflow not being prioritized and scheduled
> -----------------------------------------------------------
>
> Key: AIRFLOW-84
> URL: https://issues.apache.org/jira/browse/AIRFLOW-84
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Kevin Lin
>
> We consistently hit an issue that Airflow didn't schedule any tasks/jobs for
> several hours until we restart worker & scheduler.
> We're on 1.6.2 with celeryd enabled.
> From scheduler logs, we didn't observe any suspicious error messages. Other
> than during the entire time period (several hours), it's always {quote}
> Prioritizing 0 queued jobs. {quote}
> While in the worker logs, we did find error messages like following. The
> message is sort of generic, so not sure anything quite useful there.
> But wondering what might cause MainProcess keeps failing in celery_executor?
> e.g. OOM on the client machine etc.?
> {quote}
> 2016-05-04T07:44:10.10441 2016-05-04 07:44:10,104: ERROR/MainProcess Task
> airflow.executors.celery_executor.execute_commanda900d1f1-6abf-4d13-a403-7ec884abd565
> raised unexpected: AirflowException('Celery command failed',)
> 2016-05-04T07:44:10.10442 Traceback (most recent call last):
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 240, in trace_task
> 2016-05-04T07:44:10.10442 R = retval = fun(*args, **kwargs)
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 438, in protected_call
> 2016-05-04T07:44:10.10443 return self.run(*args, **kwargs)
> 2016-05-04T07:44:10.10443 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
> line 43, in execute_command
> 2016-05-04T07:44:10.10443 raise AirflowException('Celery command failed')
> 2016-05-04T07:44:10.10443 AirflowException: Celery command failed
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
