jedcunningham opened a new pull request, #27725:
URL: https://github.com/apache/airflow/pull/27725
If you have a large number of paused DAGs with running DagRuns in your
instance,
the SchedulerJob function that tries to update the state for those DagRuns
can be really slow (slow enough to cause the scheduler to be killed by
k8s livenessprobes, for example).
This PR does a few things:
- Only check paused DAGs that have running DagRuns
- Only check DagRuns that have TI's updated since the last
DagRun.last_scheduling_decision
- Only get the serialized DAG if we are ready to try updating the DagRun
state
This prevents Airflow from trying to update every "paused" DagRun over
and over again, and doesn't do expensive work until we know it makes
sense to try updating the state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
