kaxil commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-701053868
Ran some benchmarks against different versions (more comprehensive benchmarks will be provided -- consider this too as **_preliminary_**): **Scenario**: 10 Dag File | 1 Dag Per File | 10 Tasks per DAG **Configs**: - Executor: CeleryExecutor - Parallelism: 128 - Workers: 4 (1 CPU - 3.75 GB memory) - Each Scheduler (1 CPU - 3.75 GB memory) **Airflow 1.10.10** TI Lag Avg: 64.845s (± 7.995) Time in Queued State: 57.6485s (± 7.395) **Airflow Master** TI Lag Avg: 48.5754s (± 62.991) Time in Queued State: 11.909s (± 8.252) **This PR** (Single Scheduler) TI Lag Avg: 10.262s (± 5.299) Time in Queued State: 9.3494s (± 5.355) ------ **Scenario**: 100 Dag Files | 1 Dag Per File | 10 Tasks per DAG **Configs**: - Executor: CeleryExecutor - Parallelism: 128 - Workers: 4 (1 CPU - 3.75 GB memory) - Each Scheduler (1 CPU - 3.75 GB memory) **Airflow 1.10.10** TI Lag Avg: 809.537s (± 29.6730) Time in Queued State: 68.708s (± 15.5567) **Airflow Master** TI Lag Avg: 392.806s (± 318.620) Time in Queued State: 10.694s (± 9.930) **This PR (with 2 Schedulers)** TI Lag Avg: 74.028s (± 17.215) Time in Queued State: 68.708s (± 15.556) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
