pavansharma36 commented on PR #27060: URL: https://github.com/apache/airflow/pull/27060#issuecomment-1309036169
@ashb Existing implementation adds new file to parsing queue only if parsing queue is empty. Consider we set refresh interval to 5 seconds and have 10k existing dag files which takes around 5 minutes to parse, Even though file paths are refreshed every 5 seconds new files are not added to parsing queue until all 10k files are parsed and queue becomes empty This change to make sure if file_path is refreshed on list_interval config, new files are added to parsing queue without waiting for existing file parsing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
