No. There is no such feature yet. Assumption is that DAGs change "slowly" and without heavy structural changes that would do any harm in those situations.
If you need "heavy" change to a DAG, you need to create a new DAG with new dag_id - this is current recommendation when it comes to "dag versioning" There is an Airflow Improvement Proposal https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning about DAG versioning where such atomic changes will likely become a reality - but it has not been worked on for quite some time (but this is is likely to change in the coming months and work on DAG Versioning will resume - or so I hear). J. On Thu, May 12, 2022 at 11:57 PM Mandeep Sandhu <[email protected]> wrote: > Hi All, > > Relative newbie here. > > Is it possible (or is there a known pattern) for allowing atomic > updates of DAGs? > > By atomic I mean, if a DAG is in progress running some version of the > code, then it should continue to use it even if the DAG code is > changed. The change should be used up only for new DAG runs started > after the update. > > Eg I have a dag setup as: > A ---> B ---> C > Each step produces an intermediate build artifact which is used as > input by the next stage. If the DAG step is currently executing "B" > and I update the DAG code, I dont want the changes to be picked up > until "C" completes. > > Does airflow controller provide any such feature or is this something > that each developer handles on their own? > > Thanks for your time. > > -mandeep >
