[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-05-31 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307316#comment-15307316
 ] 

ASF subversion and git services commented on AIRFLOW-25:


Commit 03e8e5d49f2539057574daecd3f1711e666c29e9 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=03e8e5d ]

[AIRFLOW-25] Configuration for Celery always required

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- AIRFLOW-25

For now, if airflow.cfg has no [celery] section, all subcommands fail.
This patch adds the default values for Celery-related properties
as well as existing 'default_queue' and 'flower_port',
so as to make all subcommands work and suppress "not found in config"
warnings even if [celery] section is omitted.

Author: Kengo Seki 

Closes #1558 from sekikn/AIRFLOW-25.


> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Assignee: Kengo Seki
>Priority: Trivial
>  Labels: newbie
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-05-10 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278351#comment-15278351
 ] 

Chris Riccomini commented on AIRFLOW-25:


There have already been two dupes filed of this issue. It seems to be very 
common, albeit relatively benign.

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>  Labels: newbie
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-04-30 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265553#comment-15265553
 ] 

Chris Riccomini commented on AIRFLOW-25:


Appears to exist on master, as well. It looks like the CLI makes an assumption 
about what will be in the config. This is annoying.

[~ludovicc], in the short term, I recommend just adding the defaults back into 
the file.

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-04-30 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265552#comment-15265552
 ] 

Chris Riccomini commented on AIRFLOW-25:


Occurs elsewhere in the CLI as well, e.g.:

{code}
parser_kerberos.add_argument(
"-kt", "--keytab", help="keytab",
nargs='?', default=configuration.get('kerberos', 'keytab'))
parser_kerberos.add_argument(
"principal", help="kerberos principal",
nargs='?', default=configuration.get('kerberos', 'principal'))
{code}

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-04-30 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265551#comment-15265551
 ] 

Chris Riccomini commented on AIRFLOW-25:


Looks like, when the CLI is created, it's trying to set defaults in the CLI 
from config params:

https://github.com/airbnb/airflow/blob/1.7.0/airflow/bin/cli.py#L741

{code}
parser_worker.add_argument(
"-q", "--queues",
help="Comma delimited list of queues to serve",
default=configuration.get('celery', 'DEFAULT_QUEUE'))
parser_worker.add_argument(
"-c", "--concurrency",
type=int,
help="The number of worker processes",
default=configuration.get('celery', 'celeryd_concurrency'))
{code}

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-04-30 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265550#comment-15265550
 ] 

Chris Riccomini commented on AIRFLOW-25:


Derp, NM, I see it's 1.7.0. Missed that, sorry.

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

2016-04-30 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265549#comment-15265549
 ] 

Chris Riccomini commented on AIRFLOW-25:


What version are you running?

> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Priority: Trivial
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)