hussein-awala commented on PR #28711: URL: https://github.com/apache/airflow/pull/28711#issuecomment-1371539491
@ashb I reread the code of `DagFileProcessorManager` and I found that it only does two types of cleaning: 1- clean stale dags 2- clean the import errors for files that no longer exist The two cleaning depend on a change in the dag folder, and if nothing is changed in the dags folder, there will be no dags considered as stale, and the `DagFileProcessorManager` will not delete any import error because all the files are still there. This config is useful when the dags files are changed slowly, and in order to import the new dags and update the existing dags, we need to run a standalone dag processor (at least once) which can detect the stale dags and delete them, and clean the import errors for the for files that no longer exist. > else some states in case of failure will never progress If there is another cleanup done by `DagFileProcessorManager`, please mention it to check how to handle it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
