Re: celery problem: cannot override celery_broker_transport_options

Ash, Thanks! You put me on the right track. Unfortunately, there is a lot of logic in airflow/config_templates/default_celery.py that I need, and if I was to come up with my own class to replace: celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG then I

Re: celery problem: cannot override celery_broker_transport_options

Yes, you would need to duplicate a chunk of the default_celery in your copy right now. But you can just make it have the values you want - so it would be about 10 lines in total. It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little bit out of sync in the default .cfg and the

Re: celery problem: cannot override celery_broker_transport_options

Ash, Thanks again. You are leading me on the right path! I can prepare a patch to move the ssl_ options into the celery section. What about visbility_timeout? The error I am getting is: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 160, in create

Re: celery problem: cannot override celery_broker_transport_options

Removal of sqla as a backend is mentioned in these release notes for celery 4.0: http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#features-removed-for-lack-of-funding -- Craig On Thu, May 24, 2018 at 8:32 AM Craig Rodrigues wrote: > It looks like in Celery,

Re: celery problem: cannot override celery_broker_transport_options

You are right!! This is an interesting puzzle to follow, but we are unravelling it! -- Craig On Thu, May 24, 2018 at 8:39 AM Ash Berlin-Taylor < ash_airflowl...@firemirror.com> wrote: > Kombu (a library Celery uses) 4.1.0 added it back in >

Re: celery problem: cannot override celery_broker_transport_options

Ash, According to this: http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings visibility_timeout is supported by Redis and SQS. -- Craig On Thu, May 24, 2018 at 8:07 AM Craig Rodrigues wrote: > Ash, > > Thanks again. You are leading me

Is `airflow backfill` disfunctional?

So I'm running a backfill for what feels like the first time in years using a simple `airflow backfill --local` commands. First I start getting a ton of `logging.info` of each tasks that cannot be started just yet at every tick flooding my terminal with the keyword `FAILED` in it, looking like a

Re: celery problem: cannot override celery_broker_transport_options

Sounds like https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py#L31 should be guarded in some way to only do that for a redis:// and

Problem with the scheduler?

Hi, I came about an issue with Airflow that I am wonder how to deal with it and thought I should reach out to see if you know about this issue and/or have advice on how to deal with it. I have around 6 or 7 DAGs on Airflow that are scheduled to run at different intervals. Two of them should

Re: celery problem: cannot override celery_broker_transport_options

It looks like in Celery, the documentation for sqla broker was removed: https://github.com/celery/celery/commit/79810a26a116e9881c42a14d856fa94c40fefcd8#diff-29ccf8c96d521253467909a652e6ded2 I cannot find the pull request or release notes which document this. -- Craig On Thu, May 24, 2018 at

Re: celery problem: cannot override celery_broker_transport_options

Kombu (a library Celery uses) 4.1.0 added it back in https://github.com/celery/kombu/blob/master/Changelog#L75-L99 - I _thought_ that means it's supported in Celery again...? > On 24 May 2018, at 16:34, Craig Rodrigues

Re: Problem with the scheduler?

Hi Stephane, First cron string does not look correct. It says the 2nd of each month. Check here: https://crontab.guru/#0_7_/2_*_* However second looks okay to me. It says 2(-31)/2, every second day from 2 till 31. Maybe the first string needs 1/2? Also, check this SO (sorry) answer:

Re: Improving Airflow SLAs

Hi James, I haven’t read everything or looked at the entire PR yet but one thing I wanted to ask was you state you move the SLA miss callback to the task level. In our org and I can imagine in others, we would like to have the callback stay at the DAG level so we can see if the entire pipeline

Re: Improving Airflow SLAs

Hm, not sure I understand your question. To my mind this use case already isn't possible because while the callback function is currently a DAG attribute, the SLA value has always been set on specific tasks within the DAG. Perhaps this gets obscured by the current SLA miss email reporting

Re: Problem with the scheduler?

Also note that for example when setting up monthly jobs, the job with an execution_date of `2018-02-01` will be triggered soon after the wall clock hits `2018-03-01`, and that your start_date for the tasks in the DAG need to be prior to that execution_date, not the time at which you're expecting

Re: Is `airflow backfill` disfunctional?

I had a similar experience but don't remember the details - it was necessary to delete all dag runs and tasks for items you wanted to backfill. We probably could have dropped those database rows but did not try. This was primarily for when there were connection issues or input files missing that

Re: Improving Airflow SLAs

Just giving this a bump; it's a pretty major rework so I'd love to know whether this effort is likely to be accepted if I bring it to a PR-able state, before I invest more time. On Wed, May 23, 2018 at 1:59 PM, James Meickle wrote: > Hi folks, > > I've created a branch