iw-pavan opened a new issue, #27010:
URL: https://github.com/apache/airflow/issues/27010
### Apache Airflow version
2.4.1
### What happened
When there are large number of dag files, lets say 10K and each takes
sometime to parse, dag_parser doesnt pick any newly created files till all 10k
files are finished parsing
`if not self._file_path_queue:
self.emit_metrics()
self.prepare_file_path_queue()`
Above logic only adds new files to queue when queue is empty
### What you think should happen instead
Every loop of dag processor should pick new files and add into file for
parsing queue
### How to reproduce
_No response_
### Operating System
All
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
