I am personally even super happy if Astronomer provides it to the customers with commercial obligations - until it is merged in 2.1 for example. Including the support - while we are discussing it and merging and releasing it in 2.1 (and likely later supporting migration to the community one internally).
I believe there is nothing to prevent that from the ASF rules (and community) point of view :). It just has to be transparently communicated, that's all :). J. On Fri, Sep 11, 2020 at 2:18 PM Ash Berlin-Taylor <[email protected]> wrote: > Hi Jarek, and all. > > You aren't the only one to have this thought -- it's been on my mind too. > > Sadly I wasn't able to get the code in a PR-able state before heading > off on paternity leave. I have started separating out and submitting the > "easy"/preparatory PRs to try to lessen the size of the "main" PR: > > https://github.com/apache/airflow/pull/10729 > https://github.com/apache/airflow/pull/10710 > https://github.com/apache/airflow/pull/10706 > > But yes, at some point it needs a large "big-bang" PR. That is coming. > > We will have a discussion internally and see what course we think is > best for us to take. One possible path is we submit the PR to open > discussion, and concurrently make that change available via our > astronomer images of 2.0 (which is available under Apache 2 License > without commercial obligations, so usable by anyone in the community) > > Thanks for bringing this up. > > -ash > > On Sep 11 2020, at 12:56 pm, Jarek Potiuk <[email protected]> > wrote: > > > I started to feel that we need to clarify statements about the HA > Scheduler > > for our community. Not that I am losing sleep regularly over this but it > > did keep me away last night when I started to think about it :). > > > > I have a feeling that while we already defined some - rather > > aggressive - > > timelines for 2.0, the subject of HA Scheduler was not touched in the > > previous Airflow 2.0 meetings. We are not very far from the release > > but the > > HA scheduler is implemented inside Astronomer and we have not seen any > code > > for it yet in the community. I understand that a lot of work (not only > > development but especially testing) has been put into it from the > > Astronomer team internally. > > > > I am actually quite OK with that to be like that. I think Astronomer > > is a > > super-valuable member of the community and I have no doubts Ash and Kaxil > > and Daniel and others will do an awesome job with it. I am simply afraid > > that when we see it, some of the cases that we see as needed by the > > community will require more work. This will either delay the 2.0 > > release or > > we will have to drop it from the 2.0 release. Looking at the number of > > discussions we had with - much simpler IMHO - Smart Sensors, I have the > > feeling that HA scheduler will spark even more discussions. The AIP-15 > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler > > > > was > > not very rich in detail and the last time it has been updated was March > > 2019 (!) and I have no doubt a big number of design decisions, > > observations, learning has happened in Astronomer since. > > > > And to be perfectly honest - I am ok with both of the scenarios I can > see: > > > > 1) We release HA Scheduler in 2.0 > > > > For that, I think we should start looking at the code and discuss it > > already quite some time ago IMHO. It might be too late if we want to fit > > the aggressive timeline we have - especially that there are other things > > the most active people are committing to for 2.0 and they might simply > not > > have enough time to make the quality review rounds and discussions. I > think > > we need to see it first to be even able to assess if we think we can make > > it within the timeline. > > > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer > will > > use the HA Scheduler in their own service as a "commercial" add-on or > > "advantage" of their offering. > > > > In the meantime - between 2.0 and 2.1 Astronomer could donate the code > and > > we could make sure it is reviewed and merged in the way that answers the > > needs of different community members. This has also numerous > > advantages to > > the community - similar to the case of Smart Sensors, Astronomer can test > > it in production then and solve all the teething problems of such a > > service. > > > > I cannot speak for the business models of Astronomer of course :), but it > > seems to me like a nice advantage to have for a while, from the business > > point of view. And as a community, we also benefit that we have such a > > strong member of the community with a sustainable and good business > model. > > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially > > (but also others) - Airflow would not be where it is today. And I > > would be > > 100% happy with such an approach as a PMC and member of the community > > and I > > support it a lot if Astronomer chooses this path. > > > > I think, however, it's the highest time that we decide and clearly > > communicate it to the users as a community. At least I have a feeling > that > > without the community members, committers, and some heavy users being > > involved in the open, and having time for quality review and discussion, > > releasing HA in 2.0 might be not possible. And to just reiterate - > > this has > > nothing to do with the expected quality of the code and testing, but more > > about potential differences in expectations, assumptions, understanding, > > performance limitations, and anything else that might (and usually does) > > come up. > > > > I think - since we already started to publish the schedule, this is the > > right time that we make a decision on that and align expectations. > > > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a > > company for the HA Scheduler are? I know as a group of committers we said > > it a number of times that HA Scheduler will be in 2.0 so we built the > > expectations among our users as a community. But maybe you really think > > that pursuing scenario 2) (or maybe another scenario I have not thought > > about) is the way to go for Astronomer? > > > > As I wrote above - I am personally perfectly fine with either of the > > scenarios, and I think they are both beneficial for the community, but I > > think we should discuss it, align expectations, and clearly > > communicate as > > the Apache Airflow community. > > > > J. > > > > -- > > Jarek Potiuk > > Polidea | Principal Software Engineer > > > > M: +48 660 796 129 > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
