Hi All,
We have a use case to support 1000 concurrent DAGs. These dags would have have
couple of Http task which would be submitting jobs to external services. Each
DAG could run for couple of hours.
HTTP tasks are periodically checking(with sleep 20) the job status.
We tried running 1000 such dags(Parallelism set to 1000) with Airflow's
LocalExecutor Mode but after 100 concurrent runs, tasks started failing due to
--> OOM error
--> Scheduler marked them failed because of lack of heartbeat.
We are using 4 cores and 16 GB RAM. Each airflow worker is taking ~250 MB of
Virtual memory and ~60 MB of RES memory which seems to be on higher side. CPU
utilisation is also ~98%.
Is there anything that can be done to optimise Memory/CPU for airflow worker.
Any pointer to airflow benchmarking with LocalExecutor would also be helpful
