ashb opened a new issue #16366: URL: https://github.com/apache/airflow/issues/16366
There have been various problems with the behaviour of max_active_runs not being respected (https://github.com/apache/airflow/issues/9975, https://github.com/apache/airflow/issues/14744, and we should try and fix this once and for all by changing how dag runs are created and run. Currently creating a DagRun in the webserver puts it immediately in to "running" state, which can end up with too many DagRuns being executed, causing problems for the scheduler. Clearing a DagRin can have similar problems. We should change the behaviour as follows: - If the webserver creates/clears and DagRun and would previously put it in to running state, it should put it in to "queued" instead - Make SchedulerJob._create_dagruns_for_dags _also_ create DRs in queued state. - Add a new stage to SchedulerJob._do_scheduling that checks For any queued DagRun if there is "space" to promote it to running - We can remove any checks about currently_active_runs inside _schedule_dag_run function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
