MatthewStrickland opened a new issue, #41835: URL: https://github.com/apache/airflow/issues/41835
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.8.1 ### What happened? Task with higher priority (when **priority weight** _upstream_ is set) not selected to run when different DAGs are racing each other ### What you think should happen instead? Once a task has finished the executor should give a chance for the tasks of other DAGs to queue up before choosing the next task to run. (I'm assuming this is the problem why the executor sometimes chooses the wrong task to run). And/or the order of tasks should be somewhat consistent/deterministic. Fwiw, I don't ever see this issue inside a single DAG where dagruns are racing each other, just between dagruns of different DAGs. There's no mention in the docs that this feature is only consistent between dagruns of the same DAG. It's a feature coupled with pools I believe, which work across DAGs, so I assumed this would work across DAGs too. ### How to reproduce Create 2 dags (`dag 1 & 2`) and set **priority weight** _upstream_ on both Create a pool Have 5 tasks each, where both use the same pool for `tasks 3 & 4` Have `task 3` take a long time (sleep) Run both dags together **Eventually:** `dag 1` is _running_ `task 3` and `dag 2` has _queued_ `task 3` **Sometimes (and desired):** `dag 1` `task 4` runs before `dag 2` `task 3` **Sometimes (and undesired):** `dag 2` `task 3` runs before `dag 1` `task 4` ### Operating System rhel 9 ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? So this is very minor and might not even be considered a bug at all, but I found it rather unintuitive, so if it's not classed as an bug I'm ok with that, just information I thought worth sharing. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
