potiuk commented on issue #29983: URL: https://github.com/apache/airflow/issues/29983#issuecomment-1460615559
This is expected behaviour. It's inherent in the way how currently DAG folder is used as "current" source of truth even if the structure of the DAG effectively changed in the meantime when task is being executed using the previous structure. This is only one of many manifestations of it. Until we discuss, approve and implement [DAG versioning](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning) that is. Any help, initiaive in defining, discussin and implementing it is most welcome. There are multiple other cases where such breaking changes in DAG can cause more problems like that, solving (or masking) just one of those those problems makes very little sense. we need to make DAG versioning work to allow users to not care about their DAGs being parseable. It makes the problem even worse because people might get used to the fact that the DAGs starts reacting on being modified along the way. Right now the solution is "Don't change the structure of the DAG in breaking ways" (including making dynamic changes that will cause different - missing - structure when parsed at different times). Converting it into discussion in case more discussion is needed, but IMHO there is not much point in raising this and similar issues - we know it works like this - this is the current approach that is "good enough" for most cases, and DAG versioning should address the versioning issue "holistically". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
