Sounds like https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py#L31 <https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py#L31> should be guarded in some way to only do that for a redis:// and sqs:// backends.
> On 24 May 2018, at 16:13, Craig Rodrigues <rodr...@crodrigues.org> wrote: > > Ash, > > According to this: > http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings > visibility_timeout is supported by Redis and SQS. > > -- > Craig > > > On Thu, May 24, 2018 at 8:07 AM Craig Rodrigues <rodr...@crodrigues.org> > wrote: > >> Ash, >> >> Thanks again. You are leading me on the right path! >> >> I can prepare a patch to move the ssl_ options into the celery section. >> >> What about visbility_timeout? The error I am getting is: >> >> File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", >> line 160, in create >> engineclass.__name__)) >> TypeError: Invalid argument(s) >> 'ssl_key','ssl_cert','ssl_active','visibility_timeout','ssl_cacert' sent to >> create_engine(), using configuration >> MySQLDialect_mysqldb/QueuePool/Engine. Please check that the keyword >> arguments are appropriate for this combination of components. >> >> >> So it looks like visibility_timeout does not work with sqla as well. >> -- >> Craig >> >> >> On Thu, May 24, 2018 at 2:17 AM Ash Berlin-Taylor < >> ash_airflowl...@firemirror.com> wrote: >> >>> Yes, you would need to duplicate a chunk of the default_celery in your >>> copy right now. But you can just make it have the values you want - so it >>> would be about 10 lines in total. >>> >>> It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little >>> bit out of sync in the default .cfg and the celery .py - the .py is looking >>> for celery->ssl_* but the default config puts it in >>> celery_broker_transport_options->ssl_*. Looking at the celery config option >>> it looks like they aren't actually options for the other transports, it's >>> just that they don't complain about the extra options. I think the fix is >>> just to move them up to the celery section. >>> >>> -ash >>> >>> >>>> On 24 May 2018, at 07:21, Craig Rodrigues <rodr...@crodrigues.org> >>> wrote: >>>> >>>> Ash, >>>> >>>> Thanks! You put me on the right track. >>>> Unfortunately, there is a lot of logic in >>>> airflow/config_templates/default_celery.py that I need, >>>> and if I was to come up with my own class to replace: >>>> >>>> celery_config_options = >>>> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG >>>> >>>> then I would basically have to rewrite my own version of >>> default_celery.py >>>> >>>> Also, in airflow/config_templates/default_airflow.cfg, there is this: >>>> >>>> [celery_broker_transport_options] >>>> # The visibility timeout defines the number of seconds to wait for the >>>> worker >>>> # to acknowledge the task before the message is redelivered to another >>>> worker. >>>> # Make sure to increase the visibility timeout to match the time of the >>>> longest >>>> # ETA you're planning to use. Especially important in case of using >>> Redis >>>> or SQS >>>> visibility_timeout = 21600 >>>> >>>> # In case of using SSL >>>> ssl_active = False >>>> ssl_key = >>>> ssl_cert = >>>> ssl_cacert = >>>> >>>> >>>> None of those options work if using an sqla backend for celery >>> broker_url. >>>> >>>> I need to think about this, but this needs to be cleaned up before >>>> Airflow 1.10 is released. In >>> airflow/config_templates/default_airflow.cfg >>>> there is this: >>>> >>>> broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow >>>> >>>> >>>> So sqla is specified as the default broker_url for celery. >>>> A lot of people (including where I work) have used this template to set >>> up >>>> Airflow + Celery, and even though sqla is an "experimental" broker_url, >>>> it actually works pretty well. >>>> >>>> Now in airflow 1.10, something that was very easy to set up is now >>> really >>>> complicated and unintuitive. >>>> >>>> Would it be OK to change the airflow code so that >>>> in airflow/config_templates/default_airflow.cfg, all the options in >>>> [celery_broker_transport_options] are commented out? >>>> And if someone is running Redis, they would have to add those >>>> options in their own airflow.cfg file? >>>> >>>> Bolke, do you have any comments? >>>> >>>> -- >>>> Craig >>>> >>>> On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor < >>>> ash_airflowl...@firemirror.com> wrote: >>>> >>>>> To use with the SQLA backend to celery you need to override the options >>>>> Airflow passes to Celery. Those come from >>>>> >>> https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py >>>>> >>>>> Since you don't want most/all of those options (and there is no way in >>> the >>>>> config file to _remove_ a setting) you will have to point airflow to a >>>>> different file for the celery config: >>>>> >>>>> This line in the config is what you will need to change: >>>>> >>>>> # Import path for celery configuration options >>>>> celery_config_options = >>>>> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG >>>>> >>>>> If you create something like config/celery_config.py containing: >>>>> >>>>> >>>>> CELERY_CONFIG = { >>>>> # Just the options you want to set >>>>> } >>>>> >>>>> >>>>> (config/ should exist along side your dags/ folder, and I think it >>> should >>>>> be added to the python path already). You can then set this in the >>> config: >>>>> >>>>> celery_config_options = celery_config.CELERY_CONFIG >>>>> >>>>> That should give you complete control >>>>> >>>>> >>> >>>
