jedcunningham opened a new pull request, #60124: URL: https://github.com/apache/airflow/pull/60124
Renamed `add_files_to_queue` to `_add_new_files_to_queue` in `DagFileProcessorManager` to reduce confusion with `_add_files_to_queue` and better reflect its internal usage for newly discovered files. We also call the method only after a bundle has refreshed - we can't find new files without that, so doing it in every loop is wasteful. The method now checks `_processors` in addition to `_file_stats` before adding files. This prevents a race condition where files currently being processed (which don't yet have stats) were erroneously re-added to the parsing queue. The method also now results in the `dag_processing.file_path_queue_size` gauge being emitted after adding new files to the queue, and reduces log noise by having a single log line vs one per file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
