Hello, Will reliance on the database cause problems with MySQL? A lot of my users use this database. I am afraid that the lock mechanism in MySQL is much less stable and predictable than PostgresSQL and this can cause various stability problems. I know that Astronomer uses PostgreSQL, but Airflow supports RDMS in a production environment and both must work properly in this AIP.
Best regards, Kamil On Tue, Mar 3, 2020 at 12:50 PM Kaxil Naik <[email protected]> wrote: > > Good work on the Proposal Ash & Vikram. > > > > On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka <[email protected]> > wrote: > > > Team, > > > > > > > > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better > > Scheduling Performance' on Confluence and would very much appreciate > > feedback and suggestions from the community. > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > > > > > > > > The original AIP was filed by Xiaodong Deng on March 2nd, 2019 and has > > stalled after a while, so with his blessing, we are taking the baton on > > this AIP. We at Astronomer have heard several enterprises ask for both High > > Availability as well as greater scalability, specifically around starting > > hundreds and thousands of tasks in a very short time window. > > > > > > > > We would like to attempt this based on our experience running Airflow as a > > Service and deploying Airflow at enterprises around the globe. We believe > > that this will benefit Airflow and fuel greater adoption of Airflow for > > production pipelines within enterprises. > > > > > > > > Building on the original AIP, we have proposed an active/active model, > > where we can scale schedulers, but are staying away from the quorum > > approach. Xiaodong Deng had put in some really good thinking about the > > problem including approaches towards reducing contention between multiple > > schedulers and we have included some of those concepts here. Additional > > commenters had discussed the possibilities of leader selection and those > > challenges, and we have incorporated their thinking as well. . > > > > > > > > Any feedback, suggestions, and comments would be greatly appreciated. > > > > > > > > Best Regards, > > > > > > Ash Berlin-Taylor and Vikram Koka > >
