ashb commented on code in PR #35243:
URL: https://github.com/apache/airflow/pull/35243#discussion_r1375310439
##########
airflow/jobs/scheduler_job_runner.py:
##########
@@ -1064,12 +1063,8 @@ def _do_scheduling(self, session: Session) -> int:
callback_tuples = self._schedule_all_dag_runs(guard, dag_runs,
session)
# Send the callbacks after we commit to ensure the context is up to
date when it gets run
- # cache saves time during scheduling of many dag_runs for same dag
- cached_get_dag: Callable[[str], DAG | None] = lru_cache()(
- partial(self.dagbag.get_dag, session=session)
- )
for dag_run, callback_to_run in callback_tuples:
- dag = cached_get_dag(dag_run.dag_id)
+ dag = dag_run.dag or self.dagbag.get_dag(dag_run.dag_id,
session=session)
Review Comment:
For instance, just before this loop are these two calls:
```python
dag_runs =
self._get_next_dagruns_to_examine(DagRunState.RUNNING, session)
# Bulk fetch the currently active dag runs for the dags we are
# examining, rather than making one query per DagRun
callback_tuples = self._schedule_all_dag_runs(guard, dag_runs,
session)
```
Both of those get the dag out of the dagbag which weren't affected by an LRU
cache, but every dagrun we have here must have been in the call to
`_schedule_all_dag_runs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
