[jira] [Comment Edited] (AIRFLOW-1979) Redis celery backend not work on 1.9.0 (configuration is ignored)
[
https://issues.apache.org/jira/browse/AIRFLOW-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562524#comment-16562524
]
Sean Byrne edited comment on AIRFLOW-1979 at 7/30/18 9:21 PM:
--
I was content with the workaround but started digging into this today again
after your comment.
DEFAULT_CELERY_CONFIG is a dictionary with lowercase keys. I changed the keys
to uppercase config options and passed this through to the Celery constructor
and it worked. It looks like celery only supports lowercase keys in version 4.0
and above.
http://docs.celeryproject.org/en/latest/userguide/configuration.html#new-lowercase-settings
was (Author: seanbyrne88):
I was content with the workaround but started digging into this today again
after your comment.
DEFAULT_CELERY_CONFIG is a dictionary with lowercase keys. I changed the keys
to uppercase config options and passed this through to the Celery constructor
and it worked. It looks like celery only supports lowercase keys in version 4.0
and above.
> Redis celery backend not work on 1.9.0 (configuration is ignored)
> -
>
> Key: AIRFLOW-1979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1979
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery, worker
>Affects Versions: 1.9.0
>Reporter: Norio Akagi
>Priority: Major
>
> Worker tries to connect to RabbigMQ based on a default setting and shows an
> error as below:
> {noformat}
> [2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> [2018-01-09 16:45:43,140: WARNING/MainProcess]
> /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
> [2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to
> amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
> Trying again in 2.00 seconds...
> {noformat}
> I deploy Airflow on kubernetes so each component (web, scheduler, worker, and
> flower) is containerized and distributed among nodes. I set
> {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}}
> and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be
> seen when I run {{printenv}} in a container, but it looks completely ignored.
> Moving these values to {{airflow.cfg}} doesn't work either.
> It worked just perfectly 1.8 and suddenly stopped working when I upgraded
> Airflow to 1.9.
> Do you have any idea what may cause this configuration issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1979) Redis celery backend not work on 1.9.0 (configuration is ignored)
[
https://issues.apache.org/jira/browse/AIRFLOW-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360664#comment-16360664
]
Yuliya Volkova edited comment on AIRFLOW-1979 at 2/12/18 12:42 PM:
---
[~redtree1112], hello, version you used is form github/master branch? or from
official release prom pypi?
seems like configuration variables are not visible for airflow on start airflow
commands
was (Author: xnuinside):
[~redtree1112], hello, version you used is form github/master branch? or from
official release prom pypi?
seems like configuration variables are not visible for airflow worker
> Redis celery backend not work on 1.9.0 (configuration is ignored)
> -
>
> Key: AIRFLOW-1979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1979
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery, worker
>Affects Versions: 1.9.0
>Reporter: Norio Akagi
>Priority: Major
>
> Worker tries to connect to RabbigMQ based on a default setting and shows an
> error as below:
> {noformat}
> [2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> [2018-01-09 16:45:43,140: WARNING/MainProcess]
> /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
> [2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to
> amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
> Trying again in 2.00 seconds...
> {noformat}
> I deploy Airflow on kubernetes so each component (web, scheduler, worker, and
> flower) is containerized and distributed among nodes. I set
> {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}}
> and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be
> seen when I run {{printenv}} in a container, but it looks completely ignored.
> Moving these values to {{airflow.cfg}} doesn't work either.
> It worked just perfectly 1.8 and suddenly stopped working when I upgraded
> Airflow to 1.9.
> Do you have any idea what may cause this configuration issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1979) Redis celery backend not work on 1.9.0 (configuration is ignored)
[
https://issues.apache.org/jira/browse/AIRFLOW-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360664#comment-16360664
]
Yuliya Volkova edited comment on AIRFLOW-1979 at 2/12/18 12:40 PM:
---
[~redtree1112], hello, version you used is form github/master branch? or from
official release prom pypi?
seems like configuration variables are not visible for airflow worker
was (Author: xnuinside):
[~redtree1112], hello, did you see
[https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#celery-config]
?
{{AIRFLOW__CELERY__CELERY_RESULT_BACKEND could not be in 1.9.0, it was changed
from CELERY_RESULT_BACKEND to {{result_backend
> Redis celery backend not work on 1.9.0 (configuration is ignored)
> -
>
> Key: AIRFLOW-1979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1979
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery, worker
>Affects Versions: 1.9.0
>Reporter: Norio Akagi
>Priority: Major
>
> Worker tries to connect to RabbigMQ based on a default setting and shows an
> error as below:
> {noformat}
> [2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> [2018-01-09 16:45:43,140: WARNING/MainProcess]
> /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
> [2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to
> amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
> Trying again in 2.00 seconds...
> {noformat}
> I deploy Airflow on kubernetes so each component (web, scheduler, worker, and
> flower) is containerized and distributed among nodes. I set
> {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}}
> and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be
> seen when I run {{printenv}} in a container, but it looks completely ignored.
> Moving these values to {{airflow.cfg}} doesn't work either.
> It worked just perfectly 1.8 and suddenly stopped working when I upgraded
> Airflow to 1.9.
> Do you have any idea what may cause this configuration issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
