Alex Guziel created AIRFLOW-1133:
------------------------------------
Summary: More tasks than the concurrency limit can run
Key: AIRFLOW-1133
URL: https://issues.apache.org/jira/browse/AIRFLOW-1133
Project: Apache Airflow
Issue Type: Bug
Reporter: Alex Guziel
There are two ways to put checks on dag concurrency:
1) The scheduler not queueing too many tasks (via checking the amount of tasks
running)
2) The worker checking that not too many tasks are running (via the db)
Right now, both have issues.
1 doesn't considered queued tasks which may not be running now, but will be
running soon. Hopefully, check 2 should catch it, but it does not check the
condition properly as it only locks the row, and it seems locking the dag would
also be expensive.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
