HardikVijayPatel opened a new issue #21951:
URL: https://github.com/apache/airflow/issues/21951
### Apache Airflow version
2.2.3
### What happened
What i see happening is that dag tasks are waiting in scheduled state for
another dag scheduled first to complete all it's scheduled tasks before the
later triggered tasks can proceed even though the parallelism and
dag_concurrency is set to a higher value.
I have the following configuration set in airflow for concurrency:
parallelism = 64
max_active_tasks_per_dag = 16
dag_concurrency = 32
consider 2 dags, A and B.
dag A consists of 500 tasks which can be triggered in parallel with out
dependency on other tasks
dag B consists of 2 tasks, serial
below is the senario:
DAG A triggered first, all 500 get scheduled, and get to running 16 at a
time (due to max_active_tasks_per_dag).
Next, DAG B is triggered and the task gets scheduled, but DAG B waits for
all 500 of DAG A to complete before moving to running state even though the
parallelism and fag concurrency is set higher
### What you expected to happen
since the total number of active dags are less than the "dag_concurrency"
value, the tasks of DAG B should move to running state without having to wait
on DAG A scheduled tasks to complete
### How to reproduce
by having multiple dags running one after the other with the one dag having
more number tasks than allowed by "max_active_tasks_per_dag" followed by other
dags
### Operating System
Linux
### Versions of Apache Airflow Providers
pip freeze | grep apache-airflow-providers
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.2
apache-airflow-providers-imap==2.1.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-sqlite==2.0.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
