Hey all, We are considering deployment of airflow as our new pipeline framework. I have found out multiple articles explaining deployment of airflow in distributed environment (e.g. [1]). Unfortunately, I was not able to find out any use case where scheduler is deployed distributed on multiple nodes. Is it possible to have scheduler distributed on multiple nodes to prevent single point of failure? I haven't found any mention about it in documentation. I have found out in [2] that it is not possible but on the other hand in [3] is reference that this can be solved in new version of airflow.
Thanks, [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/ [2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME [3] https://issues.apache.org/jira/browse/AIRFLOW-678 Matúš Valo - Senior Software Engineer Ness Digital Engineering NESS KE, s.r.o., Moldavská cesta 10/B, 040 11 Košice, Slovak Republic O: +421 55 326 24 10 [email protected]<mailto:[email protected]> nesskdc.sk<http://www.nesskdc.sk/> | ness.com<http://www.ness.com/> | LinkedIn<https://www.linkedin.com/company/3050495> | Twitter<https://twitter.com/ness_tech> | Facebook<http://www.facebook.com/nesskdc/> ____________________________ Powered By Office365 The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by forwarding this email to [email protected]<mailto:[email protected]> and then delete it from your system. Ness technologies is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
