Just a note here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651> is the correct link for the AIP proposed for Scheduler HA. The other link was an old proposal from someone else.
On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <[email protected]> wrote: > I started to feel that we need to clarify statements about the HA Scheduler > for our community. Not that I am losing sleep regularly over this but it > did keep me away last night when I started to think about it :). > > I have a feeling that while we already defined some - rather aggressive - > timelines for 2.0, the subject of HA Scheduler was not touched in the > previous Airflow 2.0 meetings. We are not very far from the release but the > HA scheduler is implemented inside Astronomer and we have not seen any code > for it yet in the community. I understand that a lot of work (not only > development but especially testing) has been put into it from the > Astronomer team internally. > > I am actually quite OK with that to be like that. I think Astronomer is a > super-valuable member of the community and I have no doubts Ash and Kaxil > and Daniel and others will do an awesome job with it. I am simply afraid > that when we see it, some of the cases that we see as needed by the > community will require more work. This will either delay the 2.0 release or > we will have to drop it from the 2.0 release. Looking at the number of > discussions we had with - much simpler IMHO - Smart Sensors, I have the > feeling that HA scheduler will spark even more discussions. The AIP-15 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler > > > was > not very rich in detail and the last time it has been updated was March > 2019 (!) and I have no doubt a big number of design decisions, > observations, learning has happened in Astronomer since. > > And to be perfectly honest - I am ok with both of the scenarios I can see: > > 1) We release HA Scheduler in 2.0 > > For that, I think we should start looking at the code and discuss it > already quite some time ago IMHO. It might be too late if we want to fit > the aggressive timeline we have - especially that there are other things > the most active people are committing to for 2.0 and they might simply not > have enough time to make the quality review rounds and discussions. I think > we need to see it first to be even able to assess if we think we can make > it within the timeline. > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer will > use the HA Scheduler in their own service as a "commercial" add-on or > "advantage" of their offering. > > In the meantime - between 2.0 and 2.1 Astronomer could donate the code and > we could make sure it is reviewed and merged in the way that answers the > needs of different community members. This has also numerous advantages to > the community - similar to the case of Smart Sensors, Astronomer can test > it in production then and solve all the teething problems of such a > service. > > I cannot speak for the business models of Astronomer of course :), but it > seems to me like a nice advantage to have for a while, from the business > point of view. And as a community, we also benefit that we have such a > strong member of the community with a sustainable and good business model. > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially > (but also others) - Airflow would not be where it is today. And I would be > 100% happy with such an approach as a PMC and member of the community and I > support it a lot if Astronomer chooses this path. > > I think, however, it's the highest time that we decide and clearly > communicate it to the users as a community. At least I have a feeling that > without the community members, committers, and some heavy users being > involved in the open, and having time for quality review and discussion, > releasing HA in 2.0 might be not possible. And to just reiterate - this has > nothing to do with the expected quality of the code and testing, but more > about potential differences in expectations, assumptions, understanding, > performance limitations, and anything else that might (and usually does) > come up. > > I think - since we already started to publish the schedule, this is the > right time that we make a decision on that and align expectations. > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a > company for the HA Scheduler are? I know as a group of committers we said > it a number of times that HA Scheduler will be in 2.0 so we built the > expectations among our users as a community. But maybe you really think > that pursuing scenario 2) (or maybe another scenario I have not thought > about) is the way to go for Astronomer? > > As I wrote above - I am personally perfectly fine with either of the > scenarios, and I think they are both beneficial for the community, but I > think we should discuss it, align expectations, and clearly communicate as > the Apache Airflow community. > > J. > > -- > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
