rodolfo-nobrega opened a new issue, #35689: URL: https://github.com/apache/airflow/issues/35689
### Description I would like to propose a new feature for Apache Airflow: the ability to assign a pool to an entire task group, enabling better control over the execution of tasks within the group. This feature would be particularly useful for ensuring that tasks within a group are executed sequentially before another group starts, especially when multiple groups of tasks are designed to run in parallel. I propose adding the functionality to assign a pool directly to a task group, which would allow all tasks within the group to be managed by the pool's slot limitations. This feature would provide a much simpler and more efficient way to ensure that tasks within a group are executed in sequence before moving on to another group, especially in cases where parallel execution of multiple groups is required. This enhancement would greatly improve the usability and flexibility of task groups in Airflow, especially for complex workflows where task sequencing and parallel execution control are crucial. Thank you for considering this feature request. ### Use case/motivation In my current workflow, I have multiple task groups structured to run in parallel, each containing a sequence of tasks that should be executed in a specific order. The task groups are similar in structure and purpose, resembling the following format: Task Group 1: a >> b >> c Task Group 2: d >> e >> f Task Group 3: g >> h >> i The desired behavior is to have two task groups running in parallel at any given time, with each group executing its tasks in the defined sequence. However, I'm encountering a challenge with the current Airflow features: When using Concurrency: Airflow schedules the first task of each group (a, d, g) to run simultaneously, followed by the second tasks of each group (b, e, h), and so on. This approach does not maintain the sequential integrity of each group. When using a Pool: Assigning all tasks in a group to a specific pool still results in the same behavior as above, where Airflow schedules one task from each group in parallel, rather than executing all tasks in a single group sequentially before starting the next group. ### Related issues Current Workarounds and Limitations: SubDAGs: One solution is to use SubDagOperator to encapsulate each task group into a SubDAG. While this approach ensures sequential execution within each group, it introduces performance and complexity issues, as SubDAGs are known to be less efficient and more complex to manage. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
