[
https://issues.apache.org/jira/browse/AIRFLOW-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724817#comment-16724817
]
Ash Berlin-Taylor commented on AIRFLOW-3532:
--------------------------------------------
So you have HAProxy running on each server proxying redis and postgres traffic
from localhost:6400 to redis-host:6400 etc? If you have problems again I'd try
without HAProxy and going direct, at least to isolate possible sources of
problems.
> Apache Airflow > 1.8 don't working with Celery 4.x and don't working Celery
> 3.x using other than amqp transport
> ---------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3532
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3532
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: 1.9.0, 1.10.0, 1.10.1
> Reporter: Valeriy
> Priority: Major
> Labels: celery
>
> I needed Airflow > 1.8 with all the necessary fixes in the cluster
> configuration.
> I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with
> working DAG's. After some time, all DAG's gone away in queued. After
> restarting the worker, the problem is solved. I tried for a long time to find
> solutions to this problem (logs in DEBUG mode showed nothing) and found a
> number of discussions, one them:
> [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
> *{color:#d04437}As a result, we conclude that Airflow does not work with
> Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
> I decided to try the Celery 3.x and damn I got an WARNING:
> {code:java}
> [2018-12-17 15:43:11,136: WARNING/MainProcess]
> /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
>
> -------------- celery@myserver v3.1.26.post2 (Cipater)
> ---- **** -----
> --- * *** * --
> Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
> -- * - **** ---
> - ** ---------- [config]
> - ** ---------- .> app:
> airflow.executors.celery_executor:0x7f0093b86470
> - ** ---------- .> transport: amqp://guest:**@localhost:5672//
> - ** ---------- .> results: disabled://
> - *** --- * --- .> concurrency: 16 (prefork)
> -- ******* ----
> --- ***** ----- [queues]
> -------------- .> default exchange=default(direct) key=default
> {code}
> Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other
> than amqp. About it already wrote here
> [http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3cjira.13129586.1515519138000.610058.1515519180...@atlassian.jira%3E]
> My Airflow config:
> {code:java}
> [celery]
> celery_app_name = airflow.executors.celery_executor
> worker_concurrency = 16
> worker_log_server_port = 8793
> broker_url = redis://localhost:6400/0
> result_backend = db+postgres://airflow:pass@localhost:5434/airflow
> flower_host = 0.0.0.0
> flower_url_prefix =
> flower_port = 5555
> default_queue = default
> celery_config_options =
> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
> {code}
> How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
