Good work on the Proposal Ash & Vikram.
On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka <[email protected]> wrote: > Team, > > > > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better > Scheduling Performance' on Confluence and would very much appreciate > feedback and suggestions from the community. > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > > > > The original AIP was filed by Xiaodong Deng on March 2nd, 2019 and has > stalled after a while, so with his blessing, we are taking the baton on > this AIP. We at Astronomer have heard several enterprises ask for both High > Availability as well as greater scalability, specifically around starting > hundreds and thousands of tasks in a very short time window. > > > > We would like to attempt this based on our experience running Airflow as a > Service and deploying Airflow at enterprises around the globe. We believe > that this will benefit Airflow and fuel greater adoption of Airflow for > production pipelines within enterprises. > > > > Building on the original AIP, we have proposed an active/active model, > where we can scale schedulers, but are staying away from the quorum > approach. Xiaodong Deng had put in some really good thinking about the > problem including approaches towards reducing contention between multiple > schedulers and we have included some of those concepts here. Additional > commenters had discussed the possibilities of leader selection and those > challenges, and we have incorporated their thinking as well. . > > > > Any feedback, suggestions, and comments would be greatly appreciated. > > > > Best Regards, > > > Ash Berlin-Taylor and Vikram Koka >
