kosteev opened a new issue, #39332: URL: https://github.com/apache/airflow/issues/39332
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.3 ### What happened? If DAG processor is running in a standalone mode with defined --num-runs flag ``` airflow dag-processor --num-runs 100 ``` Some DAGs disappear from Airflow until DAG processor will run again. ### What you think should happen instead? DAGs shouldn't disappear never. ### How to reproduce - run Airflow 2.7.3 with standalone DAG processor (separate from scheduler) with --num-runs flag Example: ``` airflow dag-processor --num-runs 100 ``` - deploy some DAGs - after some time, ~20 mins (once DAG processor will parse all DAG files ~50), deploy another DAG (new DAG file) - after some time observe that DAGs will start to disappear The issue is, basically, that: - DAG processor runs until it parses each file exactly num_runs times https://github.com/apache/airflow/blob/2872d370178ffed402ac15d2983dbeda8ed7902b/airflow/dag_processing/manager.py#L1245 - if a new DAG file will be added to the DAG bag during DAG processor running, this new file will have different number of parsing runs than others already existing there - at some point other DAGs (those which were already in the DAG bag) will stopped to be parsed due to hitting run limit (they will be excluded from parsing loop), see "Last Run" column in the screenshot below https://github.com/apache/airflow/blob/2872d370178ffed402ac15d2983dbeda8ed7902b/airflow/dag_processing/manager.py#L1164 - DAG processor will continue to parse DAGs until each file will be parsed exactly num_runs times - Due to the fact that DAGs are not parsed for some time, they will be removed from Airflow DAG bag  ### Operating System Ubuntu ### Versions of Apache Airflow Providers _No response_ ### Deployment Other ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
