Hi Dan / Sid, Would you be able to elaborate on the multiple scheduler setup? Curious how that would have been deployed. Was the purpose to have some kind of failover or to distribute execution of jobs?
Thanks! On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <[email protected]> wrote: > Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1 and > we didn't seem to have issues. > > On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <[email protected]> wrote: > > > While I agree with the assessment of Sid that a lot has changed and we do > > not officially test on multiple schedulers, many changes were in the area > > of proper locking which benefit multiple schedulers. In addition the > tasks > > themselves have built in checks that they don’t run twice at the same > time. > > > > Yet YMMV. > > > > Bolke > > > > > On 24 Feb 2017, at 03:13, siddharth anand <[email protected]> wrote: > > > > > > I did run 2 or more schedulers with Local Executors up until mid last > > > year. There have been enough changes to the code and feature additions > > that > > > I don't think this is a recommended practice at this point. Also, there > > is > > > not a lot of synchronization in the scheduler to ensure this will work. > > > > > > -s > > > > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <[email protected]> > wrote: > > > > > >> Hi all, > > >> > > >> > > >> > > >> I am considering deployment of airflow as pipeline framework. I have > > found > > >> out multiple articles explaining deployment of airflow in distributed > > >> environment (e.g. [1]). Unfortunately, I was not able to find out any > > use > > >> case where scheduler is deployed distributed on multiple nodes. Is it > > >> possible to have scheduler distributed on multiple nodes to prevent > > single > > >> point of failure? I haven’t found any mention about it in > > documentation. I > > >> have found out in [2] that it is not possible but on the other hand in > > [3] > > >> is reference that this can be solved in new version of airflow. > > >> > > >> > > >> > > >> Thanks, > > >> > > >> > > >> Matus > > >> > > >> > > >> > > >> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/ > > >> > > >> [2] > https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME > > >> > > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678 > > >> > > > > >
