[
https://issues.apache.org/jira/browse/AIRFLOW-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277715#comment-15277715
]
Kevin Lin commented on AIRFLOW-84:
----------------------------------
[~jlowin] Thanks for the response! In the above logs, the failure happened in
MainProcess task, not a particular worker. Does that mean the entire airflow
was not able to start and/or crashing?
Since the error message was too generic, do you know if there are other common
reasons causing the celery workers fail? And for OOM, are there any leads to
reduce celery memory consumption?
> Queued tasks in Airflow not being prioritized and scheduled
> -----------------------------------------------------------
>
> Key: AIRFLOW-84
> URL: https://issues.apache.org/jira/browse/AIRFLOW-84
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Kevin Lin
>
> We consistently hit an issue that Airflow didn't schedule any tasks/jobs for
> several hours until we restart worker & scheduler.
> We're on 1.6.2 with celeryd enabled.
> From scheduler logs, we didn't observe any suspicious error messages. Other
> than during the entire time period (several hours), it's always {quote}
> Prioritizing 0 queued jobs. {quote}
> While in the worker logs, we did find error messages like following. The
> message is sort of generic, so not sure anything quite useful there.
> But wondering what might cause MainProcess keeps failing in celery_executor?
> e.g. OOM on the client machine etc.?
> {quote}
> 2016-05-04T07:44:10.10441 2016-05-04 07:44:10,104: ERROR/MainProcess Task
> airflow.executors.celery_executor.execute_commanda900d1f1-6abf-4d13-a403-7ec884abd565
> raised unexpected: AirflowException('Celery command failed',)
> 2016-05-04T07:44:10.10442 Traceback (most recent call last):
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 240, in trace_task
> 2016-05-04T07:44:10.10442 R = retval = fun(*args, **kwargs)
> 2016-05-04T07:44:10.10442 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
> line 438, in protected_call
> 2016-05-04T07:44:10.10443 return self.run(*args, **kwargs)
> 2016-05-04T07:44:10.10443 File
> "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
> line 43, in execute_command
> 2016-05-04T07:44:10.10443 raise AirflowException('Celery command failed')
> 2016-05-04T07:44:10.10443 AirflowException: Celery command failed
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
