potiuk commented on issue #24701: URL: https://github.com/apache/airflow/issues/24701#issuecomment-1168884173
Tehis is rather a feature than a bug and it is next to impossible to fix it, because in general case each DAG is parsed separately from each other. We are not likely going to fix it - because of dynamic nature of dag id generation, and continious parsing/re-parsing mutliple dags and this is happen in "aysynchronous way" we neverk know when the snapshot of our DAGs is complete. We never process DAGs from file as "complete list". We are parsing them in batches, and you do not have "snapshots" of all the directory available. For example if you have a DAG that moves from one file to another, even if you see second dag, it might be that either of the two dags have been already removed and will disappear next time when you get there, but if you have many dags, this parsing is continuous and you are never sure what is the "complete snapshot". Parsing of the DAGs take time and you cannot do it "instantly" for all DAGs to keep consistent snapshot. The current approach is a compromise - such clashes are possible, but it also makes it possible to handle cases where there are huge folders that you won't pay performance penalty of snapshotting the whole directory with all DAG files and making sure tha this snapshot is consistent. It's just price to pay for speed. But if you hve an idea how to implement it, feel free to continue discussing it - maybe you can find a good solution to that. Converting it into discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
