vitaly-krugl edited a comment on issue #12348: URL: https://github.com/apache/airflow/issues/12348#issuecomment-727083189
@ashb - The reason I was looking at `scheduler_heartbeat_sec` is that I am trying to improve the performance of system-level tests in my airflow-based app. What I am seeing is that even under very low utilization, Airflow adds latency of 4-5 seconds for executing each task. I was looking at airflow.cfg options to tune to eliminate this 4-5 second per task latency on my testing setup. I haven't been able to find any combination of options that would reduce the latency below 4-5 seconds. **Any suggestions about how to eliminate these latencies?** My test setup: one DAG with two tasks: Task A and Task B, with `A >> B` relationship. Implementation is in python. Each one of the python callbacks does minimum (almost no-op) work that shows up in the logs at < 1 sec. However, each dagrun takes upwards of 9 seconds and I observe from the timestamps (dagrun, Task A start/end and Task B start/end) that there are 3-5 second gaps between dagrun and Task A start, as well as between Task A end and Task B start. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
