Gollum999 opened a new issue, #25032: URL: https://github.com/apache/airflow/issues/25032
### Description I would like the ability to execute a group of dynamic tasks in a depth-first fashion. This is not an original idea - the [original AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42+Dynamic+Task+Mapping#AIP42DynamicTaskMapping-MapforTaskGroupsObjects) for Dynamic Task Mapping suggested using `TaskGroup`s for this feature, which seems like a great idea to me. However I haven't seen any evidence indicating that this was still a planned feature, so I just wanted to make sure that this idea is not forgotten. ### Use case/motivation Imagine a DAG like the following: ``` ┌──────────────┐ ┌────────────────┐ ┌───────────┐ ┌──►│extract_file_1│►│transform_file_1│►│load_file_1│ │ └──────────────┘ └────────────────┘ └───────────┘ │ │ ┌──────────────┐ ┌────────────────┐ ┌───────────┐ ┌─────────────┐ ├──►│extract_file_2│►│transform_file_2│►│load_file_2│ │get_file_list├─┤ └──────────────┘ └────────────────┘ └───────────┘ └─────────────┘ │ │ ... ... ... │ │ ┌──────────────┐ ┌────────────────┐ ┌───────────┐ └──►│extract_file_N│►│transform_file_n│►│load_file_N│ └──────────────┘ └────────────────┘ └───────────┘ ``` Currently, this DAG would execute in breadth-first order, i.e. all mapped task instances of the same type must be complete before moving on to the next "phase". This introduces unnecessary bottlenecks in the pipeline. Each of these "branches" is independent, so there is no need to wait for `extract_file_N` before beginning `transform_file_1`, for example. ### Related issues I couldn't find any. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
