Agreed. We love the dynamic DAG generation ourselves. But as far as I know once a DAG is created (dynamically or statically) it is best not to change it without versioning the DAG and creating a new one.
I will let others answer as to whether or not that is on the feature roadmap. > On Dec 5, 2017, at 11:22 AM, Yifei Hong <[email protected]> wrote: > > Andy, thanks much for the prompt response, this is helpful. > > So scheduler_heartbeats_sec controls how frequently the scheduler will run > and refresh the dags, right? Then in my case as we retrieve data and > dynamically generate subdags, if the current run is still undergoing and > external data source changes (i.e. from 3 run request to 5), I feel the dag > refresh will cause problem and mess up the running dag. Will Airflow > support this kind of usage? Dynamic dag generation is one of the Airflow > features that I like the most. > > On Tue, Dec 5, 2017 at 11:02 AM, Andy Cooper <[email protected]> > wrote: > >> Yifei, >> >> This is because the DAG definition file is not stored in the database. The >> scheduler must refresh the display of the DAG and it’s tasks from the DAG >> file itself. This means that it will be refreshed every time the scheduler >> is run. The schedule_interval refers to the schedule at which the DAG will >> run, not at which it will be refreshed. >> >> I would recommend not refreshing the task_ids and instead use static >> task_ids to handle an array/ list of tasks. >> >> >>> On Dec 5, 2017, at 10:58 AM, Yifei Hong <[email protected]> wrote: >>> >>> Hi, Community, >>> >>> We use the main dag in Airflow to retrieve data from a database, and then >>> dynamically create a list of subdags for parallel runs with >> LocalExecutor. >>> For some task_id, we construct from now() to ensure the uniqueness. >>> >>> One problem we encounter is that Airflow keeps refreshing the dag so that >>> the task_id gets updated accordingly, even though we did not schedule the >>> dag and only run it via backfill. >>> >>> Could you please help advise how we can control the refresh frequency? >> Why >>> does Airflow not follow the scheduled frequency to refresh? >>> >>> Many thanks in advance, >>> Yifei >> >>
