Hello Jens, I agree with everything you said, for some reason the "Deferred" state is not counted towards an active task, where intuitively it should be part of the group.
As I see it at least, all the configurations talk about *active* tasks (such as max_active_tasks, max_active_tis_per_dag, max_active_tis_per_dagrun), which I think is a quite confusing term. And to solve this, a clear explanation of what is an "active" task should be defined. It is possible to define that an "active" task is any task which is either running, queued OR deferred, but this will enforce a new configuration for backwards compatibility, such as "count_deffered_as_active" (yet this is more enforcing and global approach, which we might not want), while not introducing too much additional complexity, as adding more parameters by which we schedule tasks will only make scheduling decisions harder, as more parameters need to be checked, which will most likely slow down each decision, and might slow down the scheduler. I liked Jarek's approach, however, I think that maybe instead of introducing a few new params, we instead rename the current parameters, while keeping behavior as is, slowly deprecating the "active" configurations, as Jarek said, and for some time keep both the "active" and the "running" param, while having the "active" be an alias for "running" until the "active" is deprecated. If there is a need for a param for deferred tasks, it is possible to add only for deferrable tasks, in order to not impact current scheduling decisions done by the scheduler. I see both approaches as viable, yet I think that adding an additional param might introduce more complexity, and maybe should be split out of the regular task flow, as a deferrable task is not the same as a running task, I tend to lean towards the first approach, as it seems to be the simplest, however, the second approach might be more beneficial long-term. Best Regards, Natanel On Sun, 22 Feb 2026 at 18:43, Jarek Potiuk <[email protected]> wrote: > +1. But I think that there are cases where people wanted to > **actually** use `max_*` to limit how many workers the DAG or DAG run > will take. So possibly we should give them such an option—for example, > max_running_tis_per_dag, etc. > > There is also the question of backward compatibility. I can see the > possibility of side effects - if that changes "suddenly" after an > upgrade. For example it might mean that some Dags will suddenly start > using far fewer workers than before and become starved. > > So - if we want to change it, I think we should deprecate "_active" > and possibly add two new sets of parameters with different names—but > naming in this case is hard (more than usual). > > J. > > On Sun, Feb 22, 2026 at 5:25 PM Pavankumar Gopidesu > <[email protected]> wrote: > > > > Hi Jens, > > > > Thanks for starting this discussion. I agree that we should update how > > these tasks are counted. > > > > I previously started a PR[1] to include deferred tasks in > max_active_tasks, > > but I was sidetracked by other priorities. As you noted, this change > needs > > to encompass not only max_active_tasks but also the other parameters you > > described. > > > > [1]: https://github.com/apache/airflow/pull/41560 > > > > Regards, > > Pavan > > > > On Sun, Feb 22, 2026 at 12:43 PM constance.astronomer.io via dev < > > [email protected]> wrote: > > > > > Agreed. In my opinion, the only time we should not be counting deferred > > > tasks are for configurations that control worker slots, like number of > > > tasks that run concurrently on a celery worker, since tasks in a > deferred > > > state are not running on a worker (although you can argue that a > triggerer > > > is a special kind of worker but I digress). > > > > > > For the examples you’ve listed, deferred tasks should be part of the > > > equation since the task IS running, just not in a traditional worker. > > > > > > Thanks for bringing this up! This has been bothering me for awhile. > > > > > > Constance > > > > > > > On Feb 22, 2026, at 4:18 AM, Jens Scheffler <[email protected]> > wrote: > > > > > > > > Hi There! > > > > > > > > TLDR: In fix PR https://github.com/apache/airflow/pull/61769 we > came to > > > the point that it seems today in Airflow Core the "Deferred" state > seems to > > > be counted inconsistently. I would propose to consistently count > "Deferred" > > > into the counts of "Running". > > > > > > > > Details: > > > > > > > > * In Pools for a longer time (since PR > > > > https://github.com/apache/airflow/pull/32709) it is possible to > > > > decide whether tasks in deferred state are counted into pool > > > > allocation or not. > > > > * Before that Deferred were not counted into, which caused tasks > being > > > > in deferred potentially overwhelm backends which defesated the > > > > purpose of pools > > > > * Recently it was also seen that other limits we usually have on Dags > > > > defined as following do not consistently include deferred into > limits. > > > > o max_active_tasks - `The number of task instances allowed to run > > > > concurrently` > > > > o max_active_tis_per_dag - `When set, a task will be able to > limit > > > > the concurrent runs across logical_dates.` > > > > o max_active_tis_per_dagrun - `When set, a task will be able to > > > > limit the concurrent task instances per Dag run.` > > > > * This means at the moment defining a task as async/deferred escapes > > > > the limits > > > > > > > > Code references: > > > > > > > > * Counting tasks in Scheduler on main: > > > > > > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L190 > > > > * EXECUTION_STATES used for counting: > > > > > > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ti_deps/dependencies_states.py#L21 > > > > o Here "Deferred" is missing! > > > > > > > > Alternatives that I see: > > > > > > > > * Fix it in Scheduler consistently that limits are applied counting > > > > Deferred always in > > > > * There might be a historic reason that Deferred is not counting in - > > > > then a proper documentation would be needed - but I'd assume this > > > > un-likely > > > > * There are different opinions - then the behavior might need to be > > > > configurable. (But personally I can not see a reason for having > > > > deferred escaping the limits defined) > > > > > > > > Jens > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
