kaxil opened a new pull request, #74479: URL: https://github.com/apache/airflow/pull/74479
With `airflow standalone`, a Dag file added after startup takes up to five minutes to show up. The Dag processor only lists a bundle's files when that bundle [refreshes](https://github.com/apache/airflow/blob/bc96a14f77642cac7ea939e0837bf1bf66220f7a/airflow-core/src/airflow/dag_processing/manager.py#L694), and `[dag_processor] refresh_interval` defaults to 300 seconds, the value #45722 carried over from `[scheduler] dag_dir_list_interval`. `min_file_process_interval` does not help: it only re-parses files the processor [already knows about](https://github.com/apache/airflow/blob/bc96a14f77642cac7ea939e0837bf1bf66220f7a/airflow-core/src/airflow/dag_processing/manager.py#L1656). Standalone is the quick-start and local development path, where the loop is "write a Dag file, look for it in the UI", so a five-minute wait looks like the file was not picked up at all. Standalone now starts its components with `refresh_interval` set to `[dag_processor] bundle_refresh_check_interval` (5 seconds by default) and says so at startup: ``` standalone | Checking for new Dag files every 5 seconds (set [dag_processor] refresh_interval to change this) ``` Nothing changes outside `airflow standalone`. Measured in a breeze container (SQLite, examples off): start standalone, wait for the Dag processor's first scan, write a minimal Dag file, then poll `airflow dags list -o json` every 2 seconds until the Dag appears. | | Seconds until the new Dag is listed | |---|---| | `main` | 301.9 | | this PR | 9.0, 9.1 (two runs) | Most of the remaining 9 seconds is the poller: one `airflow dags list` call takes about 3.5 seconds in that container. In both runs the Dag processor's next scan came within 5 seconds of the file being written. **Why `bundle_refresh_check_interval` rather than a fixed number.** The processor only [checks](https://github.com/apache/airflow/blob/bc96a14f77642cac7ea939e0837bf1bf66220f7a/airflow-core/src/airflow/dag_processing/manager.py#L895) whether a bundle is due that often, so a lower `refresh_interval` changes nothing and a higher one only adds waiting. Refreshing the default Dags folder costs a directory walk, a read of each candidate file for the safe-mode `airflow`/`dag` check, and a few metadata queries; `LocalDagBundle.refresh()` itself does nothing. **Why only when the defaults are untouched.** An explicit choice wins. The override is skipped when `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL` is set (to any value), when `refresh_interval` in `airflow.cfg` differs from the default, or when `dag_bundle_config_list` is anything other than the default Dags folder. That last condition matters because refreshing a remote bundle such as `GitDagBundle` fetches from the remote, and standalone should not turn that into a fetch every 5 seconds. The `airflow.cfg` check compares with the default instead of asking whether the option is set, because the `airflow.cfg` Airflow generates on first run writes every default as a live line, `refresh_interval = 300` included. The cost: a hand-written `refresh_interval = 300` in `airflow.cfg` looks the same as the generated file and gets the standalone value. Setting it through the environment keeps it. Watching the Dags folder for file events would remove the polling delay altogether. That is a larger change to the Dag processor and is not part of this PR. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
