Shenghu Yang created AIRFLOW-113:
------------------------------------
Summary: DAG concurrency is not honored
Key: AIRFLOW-113
URL: https://issues.apache.org/jira/browse/AIRFLOW-113
Project: Apache Airflow
Issue Type: Improvement
Components: celery, executor, scheduler
Affects Versions: Airflow 1.6.2
Environment: Version of Airflow: 1.6.2
Airflow configuration: Running a Scheduler with LocalExecutor
Operating System: 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
Python Version: 2.7.6
Reporter: Shenghu Yang
This is reported in airflow github also:
https://github.com/apache/incubator-airflow/issues/1424
In our dag, we set the dag_args['concurrency'] = 8, however, when the scheduler
starts to run, we can see this concurrency is not being honored, airflow
scheduler will run up to num of the 'parallelism' (we set as 25) jobs.
What did you expect to happen?
dag_args['concurrency'] = 8 is honored, e.g. only run at most 8 jobs
concurrently.
What happened instead?
when the dag starts to run, we can see the concurrency is not being honored,
airflow scheduler will run up to the 'parallelism' (we set as 25) jobs.
Here is how you can reproduce this issue on your machine:
create a dag which contains nothing but 25 parallelized jobs.
set the dag dag_args['concurrency'] = 8
set the airflow parallelism to 25
then run: airflow scheduler
you will see all 25 jobs are scheduled to run, not 8.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
