Re: Bit confused about start_date and schedule_interval related to daily/weekly DAG

Hi Kyle, The execution_date of the DAG run will always be lagged one day for your daily DAG and one week for your weekly DAG. Under the hood, airflow will calculate the execution_date and next execution_date of the task, and only schedule the task when the current timestamp is bigger than the

Bit confused about start_date and schedule_interval related to daily/weekly DAG

I'm a bit confused with how the scheduler catches up in relation to start_date and schedule_interval. I have one dag that runs hourly: dag = DAG( dag_id='hourly_dag', start_date=days_ago(1), schedule_interval='@hourly', default_args=ARGS) When I start this DAG fresh it will catch

Re: Airflow 1.9.1 expected release?

Hi, We are currently looking to release 1.10 soon-ish. I do not think we plan on releasing 1.9.1 unless there is a vulnerability or critical bug. Best, Arthur On Wed, Apr 18, 2018 at 9:13 AM, Zsolt Tóth wrote: > Hey all, > > is there a planned (approximate) release

Airflow 1.9.1 expected release?

Hey all, is there a planned (approximate) release date for Airflow 1.9.1? Best, Zsolt

Re: Cancel a Running dag

We are exploring following approach for DAG cancellation. Please let us know if you see any issue with this 1) Set/create the xcom variable "cancel":"true". It would be set out of the band by updating the xcom Table in metadata store. 2) Operators would have the code to periodically check for

Re: Benchmarking of Airflow Scheduler with Celery Executor

On 2018/04/13 17:00:36, Maxime Beauchemin wrote: > If you're concerned about scheduler scalability I'd go with a bigger box. > The scheduler uses multiprocessing so more CPU power means more throughput. > > Also you may want to provision a beefy MySQL box to make

Re: Trouble with remote s3 logging

On 2018/04/16 19:09:12, Kyle Hamlin wrote: > This morning I tried to upgrade to the newer version of the logging config > file but I keep getting the following a TypeError for my database session. > I know my credentials are correct so I'm confused why this is happening