> > It's somehow related (but yes -it's more AIP-72 question). Mini scheduler > currently **actually** attempts to lock the DagRun table when it runs - > this is precisely what has been recently made as "optionally skipped" when > mini-scheduler could not obtain the lock immediately - because it wreak all > kind of havoc with mapped tasks: > https://github.com/apache/airflow/pull/39745 , and this is what backfill > scheduling will also attempt to lock - so I think it's very much related to > how this plays "together"
Yeah, mini-scheduler locks dag run table so that there aren't two "things" trying to schedule tasks for the same dag run. And the problem was, before we fixed it, they would wait, possibly a long time, to obtain the lock. And they don't anymore. But anyway, all of this is true now of both "normal" and "backfill" tasks and will remain so and I don't think there's much interaction with this AIP. But perhaps more importantly, I suspect mini scheduler actually goes away in airflow 3. But let's see. You mention sequencing. With the old way, it would loop through, create up to `max_active_dag_runs` runs, and then wait untill *all* of those tasks were complete before scheduling more dag runs. Now we'll be able to be more flexible, and e.g. create more as we go along, i.e. as one run finishes, create one more. I think it might make sense to just put a limit that the number of RUNNING backfill dag runs may not be more than `max_active_runs - 1`. This would ensure that there would always be room for 1 "normally scheduled" dag run. It seems like a promising idea but still sort of noodling on it. I did add a little language about scheduler process in the doc. If there are still parts of your feedback that need attention let me know and I'll try to address somehow.
