[ https://issues.apache.org/jira/browse/AIRFLOW-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360664#comment-16360664 ]
Yuliya Volkova edited comment on AIRFLOW-1979 at 2/12/18 12:40 PM: ------------------------------------------------------------------- [~redtree1112], hello, version you used is form github/master branch? or from official release prom pypi? seems like configuration variables are not visible for airflow worker was (Author: xnuinside): [~redtree1112], hello, did you see [https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#celery-config] ? {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND could not be in 1.9.0, it was changed from CELERY_RESULT_BACKEND to {{result_backend }}}} > Redis celery backend not work on 1.9.0 (configuration is ignored) > ----------------------------------------------------------------- > > Key: AIRFLOW-1979 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1979 > Project: Apache Airflow > Issue Type: Bug > Components: celery, worker > Affects Versions: 1.9.0 > Reporter: Norio Akagi > Priority: Major > > Worker tries to connect to RabbigMQ based on a default setting and shows an > error as below: > {noformat} > [2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables > from /usr/lib/python2.7/lib2to3/Grammar.txt > [2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables > from /usr/lib/python2.7/lib2to3/PatternGrammar.txt > [2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key > [celery/celery_ssl_active] not found in config > [2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor > will run without SSL > [2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor > CeleryExecutor > [2018-01-09 16:45:43,140: WARNING/MainProcess] > /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161: > CDeprecationWarning: > Starting from version 3.2 Celery will refuse to accept pickle by default. > The pickle serializer is a security concern as it may give attackers > the ability to execute any command. It's important to secure > your broker from unauthorized access when using pickle, so we think > that enabling pickle should require a deliberate action and not be > the default choice. > If you depend on pickle then you should set a setting to disable this > warning and to be sure that everything will continue working > when you upgrade to Celery 3.2:: > CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml'] > You must only enable the serializers that you will actually use. > warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED)) > [2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to > amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused. > Trying again in 2.00 seconds... > {noformat} > I deploy Airflow on kubernetes so each component (web, scheduler, worker, and > flower) is containerized and distributed among nodes. I set > {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}} > and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be > seen when I run {{printenv}} in a container, but it looks completely ignored. > Moving these values to {{airflow.cfg}} doesn't work either. > It worked just perfectly 1.8 and suddenly stopped working when I upgraded > Airflow to 1.9. > Do you have any idea what may cause this configuration issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)