[
https://issues.apache.org/jira/browse/AIRFLOW-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723176#comment-16723176
]
Ash Berlin-Taylor commented on AIRFLOW-3532:
--------------------------------------------
I can categorically say the Ariflow 1.8.2, Airflow 1.9.0 and Airflow 1.10.1
work with Celery 4 on Redis as a transport as that is how I run Airflow.
Are you running the scheduler and the workers on the same node? If not your
broker_url and result_backend will be in correct.
You have config for Flower - what does Flower show about the number of active
nodes.
What is shown in the output when you run {{airflow worker}}?
Finally on a worker node what do the following commands (run in the same
environment/as the Airflow user etc) show:
{code}
celery -A airflow.executors.celery_executor inspect ping
celery -A airflow.executors.celery_executor inspect active
celery -A airflow.executors.celery_executor inspect report
celery -A airflow.executors.celery_executor inspect stats
{code}
Also run the same commands from the scheduler - you should get similar or
identical output.
> Apache Airflow > 1.8 don't working with Celery 4.x and don't working Celery
> 3.x using other than amqp transport
> ---------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3532
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3532
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: 1.9.0, 1.10.0, 1.10.1
> Reporter: Valeriy
> Priority: Major
> Labels: celery
>
> I needed Airflow > 1.8 with all the necessary fixes in the cluster
> configuration.
> I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with
> working DAG's. After some time, all DAG's gone away in queued. After
> restarting the worker, the problem is solved. I tried for a long time to find
> solutions to this problem (logs in DEBUG mode showed nothing) and found a
> number of discussions, one them:
> [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
> *{color:#d04437}As a result, we conclude that Airflow does not work with
> Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
> I decided to try the Celery 3.x and damn I got an WARNING:
> {code:java}
> [2018-12-17 15:43:11,136: WARNING/MainProcess]
> /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
>
> -------------- celery@myserver v3.1.26.post2 (Cipater)
> ---- **** -----
> --- * *** * --
> Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
> -- * - **** ---
> - ** ---------- [config]
> - ** ---------- .> app:
> airflow.executors.celery_executor:0x7f0093b86470
> - ** ---------- .> transport: amqp://guest:**@localhost:5672//
> - ** ---------- .> results: disabled://
> - *** --- * --- .> concurrency: 16 (prefork)
> -- ******* ----
> --- ***** ----- [queues]
> -------------- .> default exchange=default(direct) key=default
> {code}
> Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other
> than amqp. About it already wrote here
> [http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3cjira.13129586.1515519138000.610058.1515519180...@atlassian.jira%3E]
> My Airflow config:
> {code:java}
> [celery]
> celery_app_name = airflow.executors.celery_executor
> worker_concurrency = 16
> worker_log_server_port = 8793
> broker_url = redis://localhost:6400/0
> result_backend = db+postgres://airflow:pass@localhost:5434/airflow
> flower_host = 0.0.0.0
> flower_url_prefix =
> flower_port = 5555
> default_queue = default
> celery_config_options =
> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
> {code}
> How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
