hterik opened a new issue, #24778: URL: https://github.com/apache/airflow/issues/24778
### Description A DAG executes from left-to-right. Upstream tasks trigger downstream tasks once completed. In this graph you can add [Branching](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html?highlight=branch#branching), deciding which downstream tasks to follow once an upstream task is completed. I find this branching concept to be a bit backwards and not utilizing the full potential of a dependency graph like a DAG. In a simple dag, the current BranchOperators are fairly trivial, for example ``` t1 >> t2 >> t3 >> resultA s1 >> s2 >> s3 >> resultB branch_op >> [t1, s1] ``` But when you get more complex dags, where some `sX` and `tX` start depending across each others path, you instead more often would like to choose between `resultA` and `resultB`, without having to know the web of dependencies of `sX` and `tY` and to avoid execution of tasks that are not necessary for the final outcome. Compare this to a conventional build-system, like `make` or `ninja`, where the execution-order and branching is decided automatically, based on the dependency graph and what final downstream tasks you asked it to build. For example "`make flashimages documentation test`" ---------------- The decision to build `resultA` or `resultB` can itself be complex and based on runtime-data, I'm thinking it need to be a task itself, where it can evaluate dag parameters and other data. In a way like a BranchOperator today. This is different from [dynamically generating dags](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html?highlight=branch#dynamic-dags), since that is only based on static configuration and can't use params or other per-dagrun information into account. Is there any way to achieve this with existing constructs? If not, is this a good idea to add and how big effort would such an implementation require? ### Use case/motivation Task-execution avoidance, computed based on dependencies already declared in DAG, chosen by what final outcome user want's to get out of individual dag run executions. With possibility to evaluate dagrun parameters and other runtime data in the decision. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
