kaxil opened a new pull request, #74835: URL: https://github.com/apache/airflow/pull/74835
After each scheduling pass, `_do_scheduling` loops over every Dag run it just examined and loads the run's Dag before handing the callback to the sink. Most passes produce no callback: one exists only when a run finishes and the Dag defines `on_success_callback` or `on_failure_callback`. The Dag load still happened for every run, and the sink then got `None` and did nothing. For a Dag in an unversioned bundle, such as the default dags folder, each of those loads queries the latest `dag_version`, because `DBDagBag` resolves runs without a `bundle_version` to the latest version. In a Postgres statement log of a single-task Dag on LocalExecutor, the scheduler ran that query 10 times per Dag run, and 3 of them (one per scheduling pass) came from this loop. For versioned bundles the load is usually an in-memory cache hit, so the saving there is small. This change skips runs whose callback is `None`, which is what the paused-Dag path in `_update_dag_run_state_for_paused_dags` already does. The "not found in serialized_dag table" error now logs only when a real callback is dropped. `_schedule_dag_run` already logs a missing Dag for every run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
