Hi Raman - First, we’d be happy to help you test this out with Airflow. Or you could do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker Engine + Docker Compose) to quickly spin up a test environment. Everything is hooked to Prometheus/Grafana to monitor how the system reacts to your workload.
-Ry CEO, Astronomer On April 12, 2018 at 12:23:46 PM, ramandu...@gmail.com (ramandu...@gmail.com) wrote: Hi All, We have requirement to run 10k(s) of concurrent tasks. We are exploring Airflow's Celery Executor for same. Horizontally Scaling of worker nodes seem possible but it can only have one active scheduler. So will Airflow scheduler be able to handle these many concurrent tasks. Is there any benchmarking number around airflow scheduler's scalability. Thanks, Raman
