DianaHov commented on issue #44336: URL: https://github.com/apache/airflow/issues/44336#issuecomment-5703482935
We hit this in production and I would like to implement it. Our case: a few hundred dynamically generated S3 downloader DAGs. Each must have at most one active *scheduled* run (they sync overlapping date ranges, so concurrent scheduled runs duplicate work), while manual reruns need to start immediately rather than queue behind the active one. `max_active_runs=1` gives the first half and breaks the second; `max_active_runs=N` gives the second and breaks the first. Proposed shape: - new optional DAG param `max_active_scheduled_runs`, backed by a nullable `DagModel` column - `max_active_runs` keeps its current meaning: the overall ceiling across all run types - `max_active_scheduled_runs` additionally caps `run_type=scheduled` only - default `None` → current behaviour exactly I have this working and tested locally against 3.3.1 across our DAGs: scheduled runs never overlap, manual runs are unaffected. One thing worth flagging: this needs to gate run *creation*, not just promotion. Gating only the promotion side leaves blocked scheduled runs accumulating in QUEUED, which isn't what people expect from a concurrency limit. On @jroachgolf84's question about API-triggered runs: those are already covered by this shape. `trigger_dag_run` creates them with `run_type=MANUAL` (only `triggered_by` distinguishes UI from REST_API), so they wouldn't be constrained by `max_active_scheduled_runs`, same as UI triggers. One design question before writing the PR: should asset-triggered runs count as scheduled? I assume not. Note that backfills already have an independent `Backfill.max_active_runs` and the running counts are already grouped by `backfill_id`, so this extends an existing concept rather than introducing a new one. Happy to put up the PR - just let me know on that point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
