Actually :) the "Valid" link was marked as "Original AIP" in the doc I linked :). I will fix it.
On Fri, Sep 11, 2020 at 2:14 PM Jarek Potiuk <[email protected]> wrote: > Ah. Thanks! It just popped as the "first" when I looked at it I will mark > it as [ARCHIVED} and move to the archive. > So my point about updates are not really valid :). It's rather detailed > and updated from what I see. > > J. > > On Fri, Sep 11, 2020 at 2:08 PM Kaxil Naik <[email protected]> wrote: > >> Just a note here: >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 >> < >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651> >> is >> the correct link for the AIP proposed for Scheduler HA. The other link was >> an old proposal from someone else. >> >> On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <[email protected]> >> wrote: >> >> > I started to feel that we need to clarify statements about the HA >> Scheduler >> > for our community. Not that I am losing sleep regularly over this but it >> > did keep me away last night when I started to think about it :). >> > >> > I have a feeling that while we already defined some - rather aggressive >> - >> > timelines for 2.0, the subject of HA Scheduler was not touched in the >> > previous Airflow 2.0 meetings. We are not very far from the release but >> the >> > HA scheduler is implemented inside Astronomer and we have not seen any >> code >> > for it yet in the community. I understand that a lot of work (not only >> > development but especially testing) has been put into it from the >> > Astronomer team internally. >> > >> > I am actually quite OK with that to be like that. I think Astronomer is >> a >> > super-valuable member of the community and I have no doubts Ash and >> Kaxil >> > and Daniel and others will do an awesome job with it. I am simply afraid >> > that when we see it, some of the cases that we see as needed by the >> > community will require more work. This will either delay the 2.0 >> release or >> > we will have to drop it from the 2.0 release. Looking at the number of >> > discussions we had with - much simpler IMHO - Smart Sensors, I have the >> > feeling that HA scheduler will spark even more discussions. The AIP-15 >> > < >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler >> > > >> > was >> > not very rich in detail and the last time it has been updated was March >> > 2019 (!) and I have no doubt a big number of design decisions, >> > observations, learning has happened in Astronomer since. >> > >> > And to be perfectly honest - I am ok with both of the scenarios I can >> see: >> > >> > 1) We release HA Scheduler in 2.0 >> > >> > For that, I think we should start looking at the code and discuss it >> > already quite some time ago IMHO. It might be too late if we want to fit >> > the aggressive timeline we have - especially that there are other things >> > the most active people are committing to for 2.0 and they might simply >> not >> > have enough time to make the quality review rounds and discussions. I >> think >> > we need to see it first to be even able to assess if we think we can >> make >> > it within the timeline. >> > >> > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer >> will >> > use the HA Scheduler in their own service as a "commercial" add-on or >> > "advantage" of their offering. >> > >> > In the meantime - between 2.0 and 2.1 Astronomer could donate the code >> and >> > we could make sure it is reviewed and merged in the way that answers the >> > needs of different community members. This has also numerous advantages >> to >> > the community - similar to the case of Smart Sensors, Astronomer can >> test >> > it in production then and solve all the teething problems of such a >> > service. >> > >> > I cannot speak for the business models of Astronomer of course :), but >> it >> > seems to me like a nice advantage to have for a while, from the business >> > point of view. And as a community, we also benefit that we have such a >> > strong member of the community with a sustainable and good business >> model. >> > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially >> > (but also others) - Airflow would not be where it is today. And I would >> be >> > 100% happy with such an approach as a PMC and member of the community >> and I >> > support it a lot if Astronomer chooses this path. >> > >> > I think, however, it's the highest time that we decide and clearly >> > communicate it to the users as a community. At least I have a feeling >> that >> > without the community members, committers, and some heavy users being >> > involved in the open, and having time for quality review and discussion, >> > releasing HA in 2.0 might be not possible. And to just reiterate - this >> has >> > nothing to do with the expected quality of the code and testing, but >> more >> > about potential differences in expectations, assumptions, understanding, >> > performance limitations, and anything else that might (and usually does) >> > come up. >> > >> > I think - since we already started to publish the schedule, this is the >> > right time that we make a decision on that and align expectations. >> > >> > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a >> > company for the HA Scheduler are? I know as a group of committers we >> said >> > it a number of times that HA Scheduler will be in 2.0 so we built the >> > expectations among our users as a community. But maybe you really think >> > that pursuing scenario 2) (or maybe another scenario I have not thought >> > about) is the way to go for Astronomer? >> > >> > As I wrote above - I am personally perfectly fine with either of the >> > scenarios, and I think they are both beneficial for the community, but I >> > think we should discuss it, align expectations, and clearly communicate >> as >> > the Apache Airflow community. >> > >> > J. >> > >> > -- >> > Jarek Potiuk >> > Polidea | Principal Software Engineer >> > >> > M: +48 660 796 129 >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
