Airflow supports what we call "slowly-changing" dynamic DAGs. Because the DAG files are frequently parsed, there is no issue in theory with generating different DAGs depending on your needs at a given moment. The issue is that you want to make sure that the same DAG is loaded by the scheduler and the executor. For example, if the scheduler loads up the DAG and sees a task called "random_task_012345", it will create an instruction for the executor to run that task. Then if the executor loads up the DAG and finds only "random_task_98765", it will cause a problem. So "slowly-changing" means the DAG can change, but do it on a timescale that won't cause conflicts when you're trying to run it.
One last note -- the web interface implicitly assumes that the DAG remains constant over time. You may see odd behavior with mutating DAGs. On Sat, Aug 27, 2016 at 8:04 PM Lance Norskog <[email protected]> wrote: > Yes! We use tables to generate some of our DAGs and it works well. What's > most important is to avoid block-copying code among DAGs. > > On Sat, Aug 27, 2016 at 12:05 PM, Vincent Poulain < > [email protected]> wrote: > > > Hello, > > First, thank you for your great job. Airflow made my job easier :) > > > > Is a dynamic generation of DAG a good practice ? As WePay explained > there : > > https://wecode.wepay.com/posts/airflow-wepay. > > > > In part of a DAG, I have *n* ECS tasks need to be launched daily, each > one > > with specific *params*. *n & params* depending on configuration which is > > daily updated. > > For now, I have just one task fetching this configuration and internally > > broadcast ECS task. > > I am wondering if Airflow would be suitable to generate DAG dynamically > > (depending on configuration) in order to split this task into those > > multiple ECS task (using ECSOperator, for instance). > > > > Thank you for your tips, > > > > > > -- > Lance Norskog > [email protected] > Redwood City, CA >
