I started to feel that we need to clarify statements about the HA Scheduler for our community. Not that I am losing sleep regularly over this but it did keep me away last night when I started to think about it :).
I have a feeling that while we already defined some - rather aggressive - timelines for 2.0, the subject of HA Scheduler was not touched in the previous Airflow 2.0 meetings. We are not very far from the release but the HA scheduler is implemented inside Astronomer and we have not seen any code for it yet in the community. I understand that a lot of work (not only development but especially testing) has been put into it from the Astronomer team internally. I am actually quite OK with that to be like that. I think Astronomer is a super-valuable member of the community and I have no doubts Ash and Kaxil and Daniel and others will do an awesome job with it. I am simply afraid that when we see it, some of the cases that we see as needed by the community will require more work. This will either delay the 2.0 release or we will have to drop it from the 2.0 release. Looking at the number of discussions we had with - much simpler IMHO - Smart Sensors, I have the feeling that HA scheduler will spark even more discussions. The AIP-15 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler> was not very rich in detail and the last time it has been updated was March 2019 (!) and I have no doubt a big number of design decisions, observations, learning has happened in Astronomer since. And to be perfectly honest - I am ok with both of the scenarios I can see: 1) We release HA Scheduler in 2.0 For that, I think we should start looking at the code and discuss it already quite some time ago IMHO. It might be too late if we want to fit the aggressive timeline we have - especially that there are other things the most active people are committing to for 2.0 and they might simply not have enough time to make the quality review rounds and discussions. I think we need to see it first to be even able to assess if we think we can make it within the timeline. 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer will use the HA Scheduler in their own service as a "commercial" add-on or "advantage" of their offering. In the meantime - between 2.0 and 2.1 Astronomer could donate the code and we could make sure it is reviewed and merged in the way that answers the needs of different community members. This has also numerous advantages to the community - similar to the case of Smart Sensors, Astronomer can test it in production then and solve all the teething problems of such a service. I cannot speak for the business models of Astronomer of course :), but it seems to me like a nice advantage to have for a while, from the business point of view. And as a community, we also benefit that we have such a strong member of the community with a sustainable and good business model. Without Astronomer's generous support, Ash, Kaxil, and Daniel especially (but also others) - Airflow would not be where it is today. And I would be 100% happy with such an approach as a PMC and member of the community and I support it a lot if Astronomer chooses this path. I think, however, it's the highest time that we decide and clearly communicate it to the users as a community. At least I have a feeling that without the community members, committers, and some heavy users being involved in the open, and having time for quality review and discussion, releasing HA in 2.0 might be not possible. And to just reiterate - this has nothing to do with the expected quality of the code and testing, but more about potential differences in expectations, assumptions, understanding, performance limitations, and anything else that might (and usually does) come up. I think - since we already started to publish the schedule, this is the right time that we make a decision on that and align expectations. Ry, Vikram - I'd love to hear what the intentions of Astronomer as a company for the HA Scheduler are? I know as a group of committers we said it a number of times that HA Scheduler will be in 2.0 so we built the expectations among our users as a community. But maybe you really think that pursuing scenario 2) (or maybe another scenario I have not thought about) is the way to go for Astronomer? As I wrote above - I am personally perfectly fine with either of the scenarios, and I think they are both beneficial for the community, but I think we should discuss it, align expectations, and clearly communicate as the Apache Airflow community. J. -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129
