Hi Airflow Devs,
as discussed in
https://lists.apache.org/thread/vvjpo6q6h0j4f80dq36g4oo0mrp5ldbq about
how deferred status of tasks is counted into running, after discussion
has "calmed down" I'd like to call for a LAZY consensus with the
following outcome:
* Short term, as some (like Bosch) is affected by deferred tasks not
counting into Dag / Task / TI concurrency we continue with Fix PR
https://github.com/apache/airflow/pull/61769
* We make the counting of deferred status a config flag so that (on a
global level) depending on needs deferred can be counted-in or not.
o Means technically: All 3 concurrency maps
"dag_run_active_tasks_map (dag_id+run_id)",
"task_concurrency_map (dag_id+task_id)" and
"task_dagrun_concurrency_map (dag_id+run_id+task_id)" will
either include deferred or not
* Defaulting to current behavior which does NOT count deferred in.
As the discussion revealed there is a general demand to configure
deferred counting on a per Dag level (like it is today with pools) we
agree on developing a mid-term concept that concurrency can be defined
with more parameters. But this would take a bit more time.
If somebody objects to this please respond to this email in the next 72h
as calling for LAZY CONSENSUS, meaning until Wednesday March 18th, 23:30
CET.
Target would be that the config flag can get into 3.2.0 as a short term
remidiation.
Jens
