Yifei, This is because the DAG definition file is not stored in the database. The scheduler must refresh the display of the DAG and it’s tasks from the DAG file itself. This means that it will be refreshed every time the scheduler is run. The schedule_interval refers to the schedule at which the DAG will run, not at which it will be refreshed.
I would recommend not refreshing the task_ids and instead use static task_ids to handle an array/ list of tasks. > On Dec 5, 2017, at 10:58 AM, Yifei Hong <[email protected]> wrote: > > Hi, Community, > > We use the main dag in Airflow to retrieve data from a database, and then > dynamically create a list of subdags for parallel runs with LocalExecutor. > For some task_id, we construct from now() to ensure the uniqueness. > > One problem we encounter is that Airflow keeps refreshing the dag so that > the task_id gets updated accordingly, even though we did not schedule the > dag and only run it via backfill. > > Could you please help advise how we can control the refresh frequency? Why > does Airflow not follow the scheduled frequency to refresh? > > Many thanks in advance, > Yifei
