potiuk edited a comment on issue #18665: URL: https://github.com/apache/airflow/issues/18665#issuecomment-932981856
This is not how Airflow works. The DAG that disappeared and appeared again wiht the same DAG is "the same" DAG. If you want to change the structure of DAG heavily you are supposed to create a new DAG with new DAG_Id and it will have new history. See the comment here: https://airflow.apache.org/docs/apache-airflow/stable/index.html?highlight=slowly#beyond-the-horizon > Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity. We have a DRAFT "Airlfow Improvement Proposal" to handle DAG versioning https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - but it needs a lot of work and discussion on how DAG versioning should look like in various cases. In fact even then when we implement it, your case will not work as you described it most likely. Airflow has no notion of files being deleted and recreated, if the DAG will appear in the same file ID it will likely never be considered as "new DAG" (this is what you effectively want). It is just the same DAG that disaappeared and appeared again. This feature is absolutely necessary to provide stability of Airflow behaviour, independently of the FileSystem behaviour. The case you describe is indistinguishable from temporary failure of the FileSystem where files will not be "visible" for some time and Airflow has to be resilient to those kind of issues. There is another AIP https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher which is closer to what you would like to achieve, but still you would have to create a new DAG this way - the old dag with history will remain as it is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
