I proposed PR for that one: https://github.com/apache/airflow/pull/21696 - if there are any comments, feel free to comment there, Otherwise I will call for a lazy consensus on that (unless there will be some doubts and concerns).
On Fri, Feb 11, 2022 at 1:01 PM Jarek Potiuk <[email protected]> wrote: > Just to conclude - I think we should consider this change when needed. And > one year seems to be fine. I think this also applies when we release > Airflow 3 (so we should support the latest 2.x line for 12 months after the > 2.x version has been released. > > One thing that we need to clarify is whether we take the FIRST or the LAST > release from the X.Y line. > > We could support providers for 12 months after the "LAST" patchlevel > released in this version: > > """ > Providers released by the community have a minimum supported version of > Airflow. The minimum version of Airflow is airflow's minor version (2.1.0, > 2.2.0 etc.) indicating that the providers might use features that appeared > in this release. By default (there could be justified exceptions) we drop > the minimum Airflow version, when 12 months passed since the last > PATCHELEVEL release for the MINOR version. > For example this means that by default we can upgrade the minimum version > of Airflow supported by providers to 2.2.0 in the first Provider's release > after 18th of September 2022 (18th of September 2021 is the date when the > last PATCHLEVEL of 2.1 (2.1.4) has been released). > """ > > Alternatively we could support the Airflow version 12 months after the > "FIRST" release of the line has been released (x.y.0). Then the statement > would be: > > """ > Providers released by the community have a minimum supported version of > Airflow. The minimum version of Airflow is airflow's minor version (2.1.0, > 2.2.0 etc.) indicating that the providers might use features that appeared > in this release. By default (there could be justified exceptions) we drop > the minimum Airflow version, when 12 months passed since the first release > for the MINOR version. > For example this means that by default we can upgrade the minimum version > of Airflow supported by providers to 2.2.0 in the first Provider's release > after 21st of May 2022 (21st of May 2021 is the date when the first version > of 2.1 (2.1.0) has been released). > """ > > I am leaning for the "FIRST". This is much more "foreseable" rule - > especially that there might be cases (critical security bugs) that we will > issue a very small "bugfix" release of (for example 2.2.5 version) even > months after the last "regular" release in this line. And I think it makes > perfect sense - the patchlevels we do, are basically already doing this - > supporting fixes to something released in the .0 version for some time. It > would make sense to do the same for providers. > > WDYT? > > On Mon, Jan 24, 2022 at 8:59 PM Kaxil Naik <[email protected]> wrote: > >> Completely agree with Elad. Targetting one year seems reasonable, however >> we should be clear that we will have exceptions when there is no other >> options. >> >> On Sun, Jan 23, 2022 at 6:29 AM Jarek Potiuk <[email protected]> wrote: >> >>> Any comments from others :) ? This is not something urgent, I think the >>> most important part here is to "catch" the moment where maintaining >>> providers to keep backwards compatibility becomes a burden. So far I >>> Thiink there are no "huge" problems. Some common utils and some name >>> changes. but I do not see anything "serious" enough to break the limits >>> >>> But maybe there are some strong points. >>> >>> J, >>> >>> On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <[email protected]> wrote: >>> >>>> Thanks Elad, >>>> >>>> Important point indeed. >>>> >>>> Agree this might be frustrating, but on the other hand we have to make >>>> sure Airflow is maintainable in the long term - this means for example >>>> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow >>>> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we >>>> will be releasing fixes to all the 'minor' lines ? We are already in >>>> the situation that we have not released a single fix to 2.0 once 2.1 >>>> is released, we have not released a single fix to 2.1 once 2.2 was >>>> released and we've never released a single fix to 2.2 once 2.3 was >>>> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot >>>> of bugs that were also present in 2.0 were released in 2.3. >>>> >>>> As I see our current "de-facto" policy is - whenever you need >>>> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a >>>> workaround - migrate to the latest 2.3. So we already encourage our >>>> users to go with the latest version (which is good). >>>> >>>> So for example releasing providers that are 2.3+ once we have 2.4 or >>>> 2.5 seems to me like a natural extension of that approach. And yeah - >>>> one of the important points of providers is that you can upgrade them >>>> independently. And it should stay like that. I would never propose a >>>> "fast" increase of min version. I think however that at some point >>>> getting such a "bump" in min-version makes sense. >>>> >>>> How often ? When? I am not sure, but I am pretty sure it should happen >>>> from time to time. >>>> >>>> However I do agree it might surprise some users that some providers >>>> will not get more fixes for the version of airflow they have, and >>>> surprise is never a good thing. >>>> >>>> Just a though maybe - what could help is just agreeing on the rules >>>> now and communicating them would be a good thing to do? >>>> >>>> Same as we agreed on Python version/K8S version policies. If we do >>>> some kind of agreement (for example - we support providers for - say 1 >>>> year after the first minor release of Airflow all providers we >>>> guarantee that all providers released will be compatible with that >>>> line? >>>> >>>> If we agree that 1 year is enough that would meaan: >>>> >>>> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was >>>> released) >>>> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was >>>> released) >>>> >>>> We can change it to 1.5 year or even 2 years if we think 1 year is >>>> enough, but I think 1 year should be pretty good. >>>> >>>> That would give a really nice cadence, it would be very >>>> straightforward rule, easy to understand and reason about, it will >>>> also give as an opportunity to periodically modernize our providers - >>>> on the other hand it would give the users a significant period where >>>> they can expect regular provider's release and enough time to prepare >>>> for bigger migration. >>>> >>>> Also this is somewhat a community-imposed upgrade cadence. Which I >>>> think is fair, reasonable and the only way we can actually (as a >>>> community) give our users certainty that they will get a good level of >>>> support from the community. It's really a trade - " You get the >>>> software from us, we agree that we will maintain it with fixes, but in >>>> return, you should follow this upgrade cadence if you want to get the >>>> full support". This is just "fair". On the other hand I think it's >>>> quite "unfair" from the users to expect from the community an infinite >>>> time of support. Making clear rules on what to expect and rules that >>>> will allow the community to provide a good level of support to the >>>> users seems like a good idea. >>>> >>>> WDYT? >>>> >>>> J. >>>> >>>> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <[email protected]> wrote: >>>> > >>>> > We need to be also mindful about the fact that not everyone is able >>>> to migrate to the new Airflow version as soon as it's being released. >>>> > One of the concepts of providers is that it's not tightly bound to >>>> Airflow core - If we can give users more time before setting a minimum >>>> version of airflow 2.3 that would be ideal. >>>> > It's frustrating when a small bug fix that you really need is beyond >>>> your reach due to some other issue that forced bumping version. It may be >>>> more hussle for us but if we can release provider bug fixes/features first >>>> and a few days later release the breaking changes in a separate release it >>>> may be valuable for some users. >>>> > >>>> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <[email protected]> >>>> wrote: >>>> >> >>>> >> One other solution for just using "utils" from core is we can >>>> release "airflow.backports" or "airflow.futures" package same as Python >>>> Backports and Python Future where we can handle the compat shim without >>>> restricting Airflow versions. >>>> >> >>>> >> Regards, >>>> >> Kaxil >>>> >> >>>> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <[email protected]> >>>> wrote: >>>> >>> >>>> >>> > At this point, I think 2.3.0 will be around early Feb to get >>>> AIP-42 in. We can reassess how the progress is moving ahead with the AIP >>>> mid-Jan and based on that we can either release 2.3.0 end of Jan without >>>> AIP-42 changes and mark it for 2.4.0. >>>> >>> >>>> >>> Yeah Agree here. I do not think we need to make a decision now - we >>>> >>> can do it later. But I'd love to hear various opinions - from our >>>> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as >>>> a >>>> >>> service for one). >>>> >>> >>>> >>> Just to be clear - I am not fully convinced myself if this is the >>>> >>> right time to do it already (but there are some good indicators that >>>> >>> we should at least start discussing it). >>>> >>> >>>> >>> This is more opening the discussion and gathering feedback/feelings >>>> of >>>> >>> people. I think such decisions should be discussed (and their >>>> >>> consequences realized) well in-advance so that at the time of >>>> decision >>>> >>> we have all arguments and opinions and know the consequences. >>>> >>> >>>> >>> And it is not "strictly" necessary. It's just (like with all >>>> similar >>>> >>> cases) an increased burden for maintenance and extra/boilerplate >>>> code >>>> >>> that might be removed. In those cases there is rarely a "0/1" >>>> decision >>>> >>> that can be made "here we know for sure that we should increase >>>> >>> min-version" - it's more of the collective thinking: "Are the >>>> >>> burden/overhead heavy enough we don't want to carry it any more as a >>>> >>> community". >>>> >>> >>>> >>> J. >>>> >>>
