potiuk commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1914532515
Airlfow does not have any "per dag" enforcement. It works pretty much exclusively on "task" level. Scheduling is not based on DAGs it's based on task. Schedulers look at tasks and picks those tasks that are eligible to run. There are thing like depends_on_pst - https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#depends-on-past that will make different instances of same task depend (and hold from being executed) on other instances, but there is no feature to have such a dependency on the whole DagRun level. You can emulate it with Pools as mentioned above, and you can write your DAGs in a way to accomodate it. Assumption is there that usually those are individual tasks that shoud not run in parallel or with too many instances, not the whole dag runs. Individual DAGs can have in-DAG limitations (such as `max_active_tasks` or you can define `max_active_runs` for it. The last case would do what you want for an indivdual DAG (i.e. Scheduler will not schedule a new DAG Run if there are are other DAG Runs of the same DAG running) If you look hard enough and think about your concrete case, (but it very much depends on nature of your limitations and why you want to exclude running in parallel and what "real" problems you want to avoid by allowing some taks from different DAGs to run in parallel) - maybe with a combination of pools/dependencies, structure of the DAGs to achieve what you want. I think in vast majority of the cases this is possible. What you are asking for @MatthewStrickland sounds like legitimate, albeit a bit niche case if you want to simplify your job of designing the DAGs / DAGRuns and think about them more like "whole entities" that can inteeract with each other - which is not the primary focus of Airlfow, because Airlfow is Task based. But - if you think it would be a valuable contribution - and something that could be usable for others as well, I think it would be fine to accept a contribution of such feature, if you you, your compan would like to contribute it - making your problem simpler to solve but also helping others in a similar situation. You are absolutely free to attempt it. Happy to review such PRs, it does not seem to be changing a lot Airflow's architecture so I do not think it needs bigger discussion or AIP, it just needs someone to roll their sleeves up and contribute it (or if those who want it do not feel strong enough to do it, even paying someone to contribute it. That would be a nice way to give back to the community and become one of the 2800 or so contributors to Airflow - many features like that were added to Airflow because the users felt the need for it and became contributors. So I heartily welcom you @MatthewStrickland and your company especially (since they are those who can likely decide on spending company resources on it) to pursue that, if the current model is not enough for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
