milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit URL: https://github.com/apache/airflow/pull/4949#discussion_r268047388
########## File path: airflow/config_templates/default_airflow.cfg ########## @@ -125,6 +125,10 @@ dags_are_paused_at_creation = True # whose size is guided by this config element non_pooled_task_slot_count = 128 +# When not using pools, the number of backfill tasks per backfill +# is limited by this config element +non_pooled_backfill_task_slot_count = 128 Review comment: Usually people don't run backfills on the machine where scheduler runs. The backfill machine can be beefier than the scheduler machine. Thus I think it make sense to allow a larger task_slot_count for backfills and that is the reason why I make it another config. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
