chsgray opened a new issue, #69187: URL: https://github.com/apache/airflow/issues/69187
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.2.1 ### What happened and how to reproduce it? Pools that include deferred tasks do not count tasks in the scheduled state. For the first scheduled state over a task lifetime, I imagine this is not problematic, but whenever there are subsequent occurrences of the scheduled state in a deferred task's lifetime, any pool tracking that task is at risk of undercounting how many slots are occupied. This reduces the efficacy of pools as a semaphore protecting some external resource, which was the motivation of https://github.com/apache/airflow/pull/32709, the PR that enabled pools to track deferred tasks. See the maintainer comment on a related issue: https://github.com/apache/airflow/issues/29416#issuecomment-1423685677. Given that this is a matter of pool implementation, reproduction is trivial. Pools will undercount by definition: [source](https://github.com/apache/airflow/blob/bfd7689273d61dc9ed9b7fc3086e29e7abbd528c/airflow-core/src/airflow/models/pool.py#L245-L249). ### What you think should happen instead? Per the [maintainer comment](https://github.com/apache/airflow/issues/29416#issuecomment-1423685677), and given that deferred tasks can transition into and out of the scheduled state arbitrarily many times, tasks in their second or greater scheduled state should count toward pool totals, provided the pool includes deferred tasks. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
