I did run 2 or more schedulers with Local Executors up until mid last year. There have been enough changes to the code and feature additions that I don't think this is a recommended practice at this point. Also, there is not a lot of synchronization in the scheduler to ensure this will work.
-s On Thu, Feb 9, 2017 at 6:47 AM, matus valo <[email protected]> wrote: > Hi all, > > > > I am considering deployment of airflow as pipeline framework. I have found > out multiple articles explaining deployment of airflow in distributed > environment (e.g. [1]). Unfortunately, I was not able to find out any use > case where scheduler is deployed distributed on multiple nodes. Is it > possible to have scheduler distributed on multiple nodes to prevent single > point of failure? I haven’t found any mention about it in documentation. I > have found out in [2] that it is not possible but on the other hand in [3] > is reference that this can be solved in new version of airflow. > > > > Thanks, > > > Matus > > > > [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/ > > [2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME > > [3] https://issues.apache.org/jira/browse/AIRFLOW-678 >
