potiuk commented on issue #17372: URL: https://github.com/apache/airflow/issues/17372#issuecomment-890881685
This problem is already solved. You should run multiple schedulers. One of the reasons Airflow HA scheduling is implemented is to allow scaling of serialization part. You can read more about HA scheduler, how it works and it's scalability characteristics in this blog post from Astronomer, https://www.astronomer.io/blog/airflow-2-scheduler#:~:text=As%20part%20of%20Apache%20Airflow,once%20their%20dependencies%20are%20met. Please try the same with multiple schedulers and let us know your experience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
