Eric Nichols created AIRFLOW-5820:
-------------------------------------
Summary: Long delay between individual tasks in a large backfill
Key: AIRFLOW-5820
URL: https://issues.apache.org/jira/browse/AIRFLOW-5820
Project: Apache Airflow
Issue Type: Improvement
Components: backfill
Affects Versions: 1.10.5
Environment: Ubuntu 18
Reporter: Eric Nichols
I am new to Airflow. I made a simple task in a trivial DAG. It takes 0.004
seconds to fill the DagBag, and the task takes only 3 seconds to run.
Max concurrency must be set to 1 since my task hits a public API with a rate
limit in effect.
I set it up to backfill 3 years of data; so I need to run the task ~1000 times
in sequence. This should take ~3000 seconds.
Unfortunately, Airflow spends 3 seconds running task, and then waits around 40
seconds before starting the next day of the backfill. So more than 90% of the
time is Airflow spinning, and the job takes ~10x longer than required.
I think there should be a way to make backfill jobs run quickly, one after
another, in this very simple case I have described. There is simply not 40
seconds worth of necessary compute to do between tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
