Just to conclude - I think we should consider this change when needed. And one year seems to be fine. I think this also applies when we release Airflow 3 (so we should support the latest 2.x line for 12 months after the 2.x version has been released.
One thing that we need to clarify is whether we take the FIRST or the LAST release from the X.Y line. We could support providers for 12 months after the "LAST" patchlevel released in this version: """ Providers released by the community have a minimum supported version of Airflow. The minimum version of Airflow is airflow's minor version (2.1.0, 2.2.0 etc.) indicating that the providers might use features that appeared in this release. By default (there could be justified exceptions) we drop the minimum Airflow version, when 12 months passed since the last PATCHELEVEL release for the MINOR version. For example this means that by default we can upgrade the minimum version of Airflow supported by providers to 2.2.0 in the first Provider's release after 18th of September 2022 (18th of September 2021 is the date when the last PATCHLEVEL of 2.1 (2.1.4) has been released). """ Alternatively we could support the Airflow version 12 months after the "FIRST" release of the line has been released (x.y.0). Then the statement would be: """ Providers released by the community have a minimum supported version of Airflow. The minimum version of Airflow is airflow's minor version (2.1.0, 2.2.0 etc.) indicating that the providers might use features that appeared in this release. By default (there could be justified exceptions) we drop the minimum Airflow version, when 12 months passed since the first release for the MINOR version. For example this means that by default we can upgrade the minimum version of Airflow supported by providers to 2.2.0 in the first Provider's release after 21st of May 2022 (21st of May 2021 is the date when the first version of 2.1 (2.1.0) has been released). """ I am leaning for the "FIRST". This is much more "foreseable" rule - especially that there might be cases (critical security bugs) that we will issue a very small "bugfix" release of (for example 2.2.5 version) even months after the last "regular" release in this line. And I think it makes perfect sense - the patchlevels we do, are basically already doing this - supporting fixes to something released in the .0 version for some time. It would make sense to do the same for providers. WDYT? On Mon, Jan 24, 2022 at 8:59 PM Kaxil Naik <[email protected]> wrote: > Completely agree with Elad. Targetting one year seems reasonable, however > we should be clear that we will have exceptions when there is no other > options. > > On Sun, Jan 23, 2022 at 6:29 AM Jarek Potiuk <[email protected]> wrote: > >> Any comments from others :) ? This is not something urgent, I think the >> most important part here is to "catch" the moment where maintaining >> providers to keep backwards compatibility becomes a burden. So far I >> Thiink there are no "huge" problems. Some common utils and some name >> changes. but I do not see anything "serious" enough to break the limits >> >> But maybe there are some strong points. >> >> J, >> >> On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <[email protected]> wrote: >> >>> Thanks Elad, >>> >>> Important point indeed. >>> >>> Agree this might be frustrating, but on the other hand we have to make >>> sure Airflow is maintainable in the long term - this means for example >>> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow >>> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we >>> will be releasing fixes to all the 'minor' lines ? We are already in >>> the situation that we have not released a single fix to 2.0 once 2.1 >>> is released, we have not released a single fix to 2.1 once 2.2 was >>> released and we've never released a single fix to 2.2 once 2.3 was >>> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot >>> of bugs that were also present in 2.0 were released in 2.3. >>> >>> As I see our current "de-facto" policy is - whenever you need >>> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a >>> workaround - migrate to the latest 2.3. So we already encourage our >>> users to go with the latest version (which is good). >>> >>> So for example releasing providers that are 2.3+ once we have 2.4 or >>> 2.5 seems to me like a natural extension of that approach. And yeah - >>> one of the important points of providers is that you can upgrade them >>> independently. And it should stay like that. I would never propose a >>> "fast" increase of min version. I think however that at some point >>> getting such a "bump" in min-version makes sense. >>> >>> How often ? When? I am not sure, but I am pretty sure it should happen >>> from time to time. >>> >>> However I do agree it might surprise some users that some providers >>> will not get more fixes for the version of airflow they have, and >>> surprise is never a good thing. >>> >>> Just a though maybe - what could help is just agreeing on the rules >>> now and communicating them would be a good thing to do? >>> >>> Same as we agreed on Python version/K8S version policies. If we do >>> some kind of agreement (for example - we support providers for - say 1 >>> year after the first minor release of Airflow all providers we >>> guarantee that all providers released will be compatible with that >>> line? >>> >>> If we agree that 1 year is enough that would meaan: >>> >>> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was >>> released) >>> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was >>> released) >>> >>> We can change it to 1.5 year or even 2 years if we think 1 year is >>> enough, but I think 1 year should be pretty good. >>> >>> That would give a really nice cadence, it would be very >>> straightforward rule, easy to understand and reason about, it will >>> also give as an opportunity to periodically modernize our providers - >>> on the other hand it would give the users a significant period where >>> they can expect regular provider's release and enough time to prepare >>> for bigger migration. >>> >>> Also this is somewhat a community-imposed upgrade cadence. Which I >>> think is fair, reasonable and the only way we can actually (as a >>> community) give our users certainty that they will get a good level of >>> support from the community. It's really a trade - " You get the >>> software from us, we agree that we will maintain it with fixes, but in >>> return, you should follow this upgrade cadence if you want to get the >>> full support". This is just "fair". On the other hand I think it's >>> quite "unfair" from the users to expect from the community an infinite >>> time of support. Making clear rules on what to expect and rules that >>> will allow the community to provide a good level of support to the >>> users seems like a good idea. >>> >>> WDYT? >>> >>> J. >>> >>> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <[email protected]> wrote: >>> > >>> > We need to be also mindful about the fact that not everyone is able to >>> migrate to the new Airflow version as soon as it's being released. >>> > One of the concepts of providers is that it's not tightly bound to >>> Airflow core - If we can give users more time before setting a minimum >>> version of airflow 2.3 that would be ideal. >>> > It's frustrating when a small bug fix that you really need is beyond >>> your reach due to some other issue that forced bumping version. It may be >>> more hussle for us but if we can release provider bug fixes/features first >>> and a few days later release the breaking changes in a separate release it >>> may be valuable for some users. >>> > >>> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <[email protected]> >>> wrote: >>> >> >>> >> One other solution for just using "utils" from core is we can release >>> "airflow.backports" or "airflow.futures" package same as Python Backports >>> and Python Future where we can handle the compat shim without restricting >>> Airflow versions. >>> >> >>> >> Regards, >>> >> Kaxil >>> >> >>> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <[email protected]> >>> wrote: >>> >>> >>> >>> > At this point, I think 2.3.0 will be around early Feb to get >>> AIP-42 in. We can reassess how the progress is moving ahead with the AIP >>> mid-Jan and based on that we can either release 2.3.0 end of Jan without >>> AIP-42 changes and mark it for 2.4.0. >>> >>> >>> >>> Yeah Agree here. I do not think we need to make a decision now - we >>> >>> can do it later. But I'd love to hear various opinions - from our >>> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as a >>> >>> service for one). >>> >>> >>> >>> Just to be clear - I am not fully convinced myself if this is the >>> >>> right time to do it already (but there are some good indicators that >>> >>> we should at least start discussing it). >>> >>> >>> >>> This is more opening the discussion and gathering feedback/feelings >>> of >>> >>> people. I think such decisions should be discussed (and their >>> >>> consequences realized) well in-advance so that at the time of >>> decision >>> >>> we have all arguments and opinions and know the consequences. >>> >>> >>> >>> And it is not "strictly" necessary. It's just (like with all similar >>> >>> cases) an increased burden for maintenance and extra/boilerplate code >>> >>> that might be removed. In those cases there is rarely a "0/1" >>> decision >>> >>> that can be made "here we know for sure that we should increase >>> >>> min-version" - it's more of the collective thinking: "Are the >>> >>> burden/overhead heavy enough we don't want to carry it any more as a >>> >>> community". >>> >>> >>> >>> J. >>> >>
