[
https://issues.apache.org/jira/browse/AIRFLOW-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289235#comment-15289235
]
Jeremiah Lowin commented on AIRFLOW-84:
---------------------------------------
I believe that the error is actually happening in the worker and being raised
in the main process -- each worker needs to load the whole Airflow environment,
so there is a non-trivial RAM requirement. I would recommend starting fewer
workers or reducing the celery concurrency in you {{airflow.cfg}} file, perhaps
all the way to 1, and seeing if that helps. Alternatively, try
{{SequentialExecutor}} as a debug mechanism to see if there is some other error
being raised.
> Queued tasks in Airflow not being prioritized and scheduled
> -----------------------------------------------------------
>
> Key: AIRFLOW-84
> URL: https://issues.apache.org/jira/browse/AIRFLOW-84
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Kevin Lin
>
> We consistently hit an issue that Airflow didn't schedule any tasks/jobs for
> several hours until we restart worker & scheduler.
> We're on 1.6.2 with celeryd enabled.
> From scheduler logs, we didn't observe any suspicious error messages. Other
> than during the entire time period (several hours), it's always {quote}
> Prioritizing 0 queued jobs. {quote}
> While in the worker logs, we did find error messages like following. The
> message is sort of generic, so not sure anything quite useful there.
> But wondering what might cause MainProcess keeps failing in celery_executor?
> e.g. OOM on the client machine etc.?
> {quote}
> 2016-05-04T07:44:10.10441 2016-05-04 07:44:10,104: ERROR/MainProcess Task
> airflow.executors.celery_executor.execute_commanda900d1f1-6abf-4d13-a403-7ec884abd565
> raised unexpected: AirflowException('Celery command failed',)
> 2016-05-04T07:44:10.10442 Traceback (most recent call last):
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 240, in trace_task
> 2016-05-04T07:44:10.10442 R = retval = fun(*args, **kwargs)
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 438, in protected_call
> 2016-05-04T07:44:10.10443 return self.run(*args, **kwargs)
> 2016-05-04T07:44:10.10443 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
> line 43, in execute_command
> 2016-05-04T07:44:10.10443 raise AirflowException('Celery command failed')
> 2016-05-04T07:44:10.10443 AirflowException: Celery command failed
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
