Alex Guziel created AIRFLOW-1133: ------------------------------------ Summary: More tasks than the concurrency limit can run Key: AIRFLOW-1133 URL: https://issues.apache.org/jira/browse/AIRFLOW-1133 Project: Apache Airflow Issue Type: Bug Reporter: Alex Guziel
There are two ways to put checks on dag concurrency: 1) The scheduler not queueing too many tasks (via checking the amount of tasks running) 2) The worker checking that not too many tasks are running (via the db) Right now, both have issues. 1 doesn't considered queued tasks which may not be running now, but will be running soon. Hopefully, check 2 should catch it, but it does not check the condition properly as it only locks the row, and it seems locking the dag would also be expensive. -- This message was sent by Atlassian JIRA (v6.3.15#6346)