Asquator commented on PR #55537: URL: https://github.com/apache/airflow/pull/55537#issuecomment-3336154399
### BENCHMARK: Workload with three limits #### Description **DAGs: 20 Tasks instances per DAG: 200 `max_active_tasks=5` Pools: 20 Slots in pool: 5 Tasks are uniformly distributed among pools TIs are divided into 40 batches of 5 mapped tasks, each one having `max_active_tis_per_dagrun=1` Operator: `BashOperator` Weight rule: downstream (default)** -> [See DAG code](https://gist.github.com/Asquator/485e4bf2e9306f4547c1855c2dbba595) <-- <details> <summary>Technical details</summary> DB: Postgres 13 CPU: 16 cores - D16s v3 (Azure) RAM: 64GB Executor: `LocalExecutor` Configurations: ``` AIRFLOW__CORE__PARALLELISM=128 AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY=16 AIRFLOW__CELERY__WORKER_AUTOSCALE=48,16 AIRFLOW__CELERY__OPERATION_TIMEOUT=60 ``` </details> #### Results | Selector strategy ➡️ <br> Measurement ⬇️| Old | New | Meaning |----------|----------|----------|----------| | Total iterations | 877 | 295 | Less by ~2.9x | | Average scheduler loop duration | 747[ms] | 1095[ms] | Longer by ~1.5x | | Total time| ~655[sec] | ~323[sec] | Throughput improvement ~2.03x On the left: **old** selector On the right: **new** selector <img width="2471" height="1377" alt="bench mapped pool" src="https://github.com/user-attachments/assets/c6baa31f-fb23-4239-8551-ede561f221e0"; /> _Note: Due to Statsd+Prometheus approximations, sometimes we see metrics at 0 when they're small enough._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
