[
https://issues.apache.org/jira/browse/AIRFLOW-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723983#comment-16723983
]
Valeriy commented on AIRFLOW-3532:
----------------------------------
[~ashb] Thanks for your reply!
{quote}Are you running the scheduler and the workers on the same node? If not
your broker_url and result_backend will be in correct.
{quote}
Yes, I'm runing airflow webserver, scheduler and worker on the same node. I
also have second active server with the same installation of Airflow. On
dedicated servers I have Redis and PostgreSQL, fault tolerance of which is
provided by the HAproxy. Since Airflow works with Redis through HAproxy, I
raised the timeouts up to 24 houres for client and server on the configuration
of HAproxy. Maybe because of the HAproxy timeouts was a problem with DAG's of
Airflow, I'll watch and to write about the results. I think what celery is so
stupid that it is not ready to re-establish TCP-session, which breaks the
receiver after timeout.
So, my Celery config:
{code:java}
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://redis-server:6400/0
result_backend = db+postgres://airflow:pass@postgres-server:5434/airflow
flower_host = 0.0.0.0
flower_url_prefix = /flower
flower_port = 5555
default_queue = default
celery_config_options =
airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
[celery_broker_transport_options]
visibility_timeout = 86400{code}
{quote}You have config for Flower - what does Flower show about the number of
active nodes.
{quote}
I,m installed Flower and again runnig DAG's in Airflow. I see a successful
tasks that handles executor of Celery, but the statistics of the broker is
zero, it may be a little later, the celery will start to put messages in Redis.
I started DAG's force and I'm waiting for them to go queued. I hope this does
not happen after changing the timeouts in the HAproxy configuration. If you
have any thoughts about the information I have provided, I will be glad to hear
your comments. If I again get the problem, then I'll proceed to the above you
diagnostic of Celery.
> Apache Airflow > 1.8 don't working with Celery 4.x and don't working Celery
> 3.x using other than amqp transport
> ---------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3532
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3532
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: 1.9.0, 1.10.0, 1.10.1
> Reporter: Valeriy
> Priority: Major
> Labels: celery
>
> I needed Airflow > 1.8 with all the necessary fixes in the cluster
> configuration.
> I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with
> working DAG's. After some time, all DAG's gone away in queued. After
> restarting the worker, the problem is solved. I tried for a long time to find
> solutions to this problem (logs in DEBUG mode showed nothing) and found a
> number of discussions, one them:
> [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
> *{color:#d04437}As a result, we conclude that Airflow does not work with
> Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
> I decided to try the Celery 3.x and damn I got an WARNING:
> {code:java}
> [2018-12-17 15:43:11,136: WARNING/MainProcess]
> /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
>
> -------------- celery@myserver v3.1.26.post2 (Cipater)
> ---- **** -----
> --- * *** * --
> Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
> -- * - **** ---
> - ** ---------- [config]
> - ** ---------- .> app:
> airflow.executors.celery_executor:0x7f0093b86470
> - ** ---------- .> transport: amqp://guest:**@localhost:5672//
> - ** ---------- .> results: disabled://
> - *** --- * --- .> concurrency: 16 (prefork)
> -- ******* ----
> --- ***** ----- [queues]
> -------------- .> default exchange=default(direct) key=default
> {code}
> Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other
> than amqp. About it already wrote here
> [http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3cjira.13129586.1515519138000.610058.1515519180...@atlassian.jira%3E]
> My Airflow config:
> {code:java}
> [celery]
> celery_app_name = airflow.executors.celery_executor
> worker_concurrency = 16
> worker_log_server_port = 8793
> broker_url = redis://localhost:6400/0
> result_backend = db+postgres://airflow:pass@localhost:5434/airflow
> flower_host = 0.0.0.0
> flower_url_prefix =
> flower_port = 5555
> default_queue = default
> celery_config_options =
> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
> {code}
> How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
