[ https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784115#comment-16784115 ]
Tao Feng commented on AIRFLOW-1557: ----------------------------------- it shouldn't be that difficult to fix. My understanding is that we need to incorporate this logic([https://github.com/apache/airflow/blob/master/airflow/jobs.py#L1106-L1207)] into backfillJob scheduler which two things need to take into consideration: 1. open slot setting based on pool; 2. task_concurrency_limit definition([https://github.com/apache/airflow/blob/master/airflow/jobs.py#L1025)] cc [~tronbabylove] [~milton0825] , did either of you want to take a stab? > backfill ignores configured number of slots in a pool > ----------------------------------------------------- > > Key: AIRFLOW-1557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1557 > Project: Apache Airflow > Issue Type: Bug > Affects Versions: 1.8.1 > Reporter: Ash Berlin-Taylor > Priority: Major > Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at > 11.39.32.png > > > I ran a backfill command with this tool: > The backfill process appears to run as many tasks as possible, even when the > pool it is running in should limit the numbers: > {noformat} > airflow backfill \ > -t fetch_dk_unfiltered \ > --pool brand_index_api \ > -s 2017-07-31 -e 2017-08-31 \ > -x \ > brand_index_fetcher > {noformat} > (Nothing else than the backfill is currently using this pool. I wasn't able > to capture a screen shot of the task instances before the jobs completed.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)