Tomasz Bartczak created AIRFLOW-137:
---------------------------------------
Summary: Airflow does not respect 'max_active_runs' when task from
multiple dag runs cleared
Key: AIRFLOW-137
URL: https://issues.apache.org/jira/browse/AIRFLOW-137
Project: Apache Airflow
Issue Type: Bug
Reporter: Tomasz Bartczak
Priority: Minor
Also requested at https://github.com/apache/incubator-airflow/issues/1442
Dear Airflow Maintainers,
Environment
Before I tell you about my issue, let me describe my Airflow environment:
Please fill out any appropriate fields:
Airflow version: 1.7.0
Airflow components: webserver, mysql, scheduler with celery executor
Python Version: 2.7.6
Operating System: Linux Ubuntu 3.19.0-26-generic Scheduler runs with
--num-runs and get restarted around every minute or so
Description of Issue
Now that you know a little about me, let me tell you about the issue I am
having:
What did you expect to happen?
After running 'airflow clear -t spark_final_observations2csv -s
2016-04-07T01:00:00 -e 2016-04-11T01:00:00 MODELLING_V6' I expected that this
task gets executed in all dag-runs in specified by given time-range -
respecting 'max_active_runs'
Dag configuration:
concurrency= 3,
max_active_runs = 2,
What happened instead?
Airflow at first started executing 3 of those tasks, which already violates
'max_active_runs', but it looks like 'concurrency' was the applied limit here.
3_running_2_pending
After first task was done - airflow scheduled all other tasks, making it 5
running dags at the same time that violates all specified limit.
In the GUI we saw red warning (5/2 Dags running ;-) )
Reproducing the Issue
max_active_runs is respected in a day-to-day basis - when of the tasks was
stuck - airflow didn't start more than 2 dags concurrently.
[screenshots in the original issue:
https://github.com/apache/incubator-airflow/issues/1442]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
