Thanks Elad, Important point indeed.
Agree this might be frustrating, but on the other hand we have to make sure Airflow is maintainable in the long term - this means for example that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we will be releasing fixes to all the 'minor' lines ? We are already in the situation that we have not released a single fix to 2.0 once 2.1 is released, we have not released a single fix to 2.1 once 2.2 was released and we've never released a single fix to 2.2 once 2.3 was released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot of bugs that were also present in 2.0 were released in 2.3. As I see our current "de-facto" policy is - whenever you need something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a workaround - migrate to the latest 2.3. So we already encourage our users to go with the latest version (which is good). So for example releasing providers that are 2.3+ once we have 2.4 or 2.5 seems to me like a natural extension of that approach. And yeah - one of the important points of providers is that you can upgrade them independently. And it should stay like that. I would never propose a "fast" increase of min version. I think however that at some point getting such a "bump" in min-version makes sense. How often ? When? I am not sure, but I am pretty sure it should happen from time to time. However I do agree it might surprise some users that some providers will not get more fixes for the version of airflow they have, and surprise is never a good thing. Just a though maybe - what could help is just agreeing on the rules now and communicating them would be a good thing to do? Same as we agreed on Python version/K8S version policies. If we do some kind of agreement (for example - we support providers for - say 1 year after the first minor release of Airflow all providers we guarantee that all providers released will be compatible with that line? If we agree that 1 year is enough that would meaan: * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was released) * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was released) We can change it to 1.5 year or even 2 years if we think 1 year is enough, but I think 1 year should be pretty good. That would give a really nice cadence, it would be very straightforward rule, easy to understand and reason about, it will also give as an opportunity to periodically modernize our providers - on the other hand it would give the users a significant period where they can expect regular provider's release and enough time to prepare for bigger migration. Also this is somewhat a community-imposed upgrade cadence. Which I think is fair, reasonable and the only way we can actually (as a community) give our users certainty that they will get a good level of support from the community. It's really a trade - " You get the software from us, we agree that we will maintain it with fixes, but in return, you should follow this upgrade cadence if you want to get the full support". This is just "fair". On the other hand I think it's quite "unfair" from the users to expect from the community an infinite time of support. Making clear rules on what to expect and rules that will allow the community to provide a good level of support to the users seems like a good idea. WDYT? J. On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <[email protected]> wrote: > > We need to be also mindful about the fact that not everyone is able to > migrate to the new Airflow version as soon as it's being released. > One of the concepts of providers is that it's not tightly bound to Airflow > core - If we can give users more time before setting a minimum version of > airflow 2.3 that would be ideal. > It's frustrating when a small bug fix that you really need is beyond your > reach due to some other issue that forced bumping version. It may be more > hussle for us but if we can release provider bug fixes/features first and a > few days later release the breaking changes in a separate release it may be > valuable for some users. > > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <[email protected]> wrote: >> >> One other solution for just using "utils" from core is we can release >> "airflow.backports" or "airflow.futures" package same as Python Backports >> and Python Future where we can handle the compat shim without restricting >> Airflow versions. >> >> Regards, >> Kaxil >> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <[email protected]> wrote: >>> >>> > At this point, I think 2.3.0 will be around early Feb to get AIP-42 in. >>> > We can reassess how the progress is moving ahead with the AIP mid-Jan and >>> > based on that we can either release 2.3.0 end of Jan without AIP-42 >>> > changes and mark it for 2.4.0. >>> >>> Yeah Agree here. I do not think we need to make a decision now - we >>> can do it later. But I'd love to hear various opinions - from our >>> users, various stakeholders (Amazon/Google/Astronomer who run it as a >>> service for one). >>> >>> Just to be clear - I am not fully convinced myself if this is the >>> right time to do it already (but there are some good indicators that >>> we should at least start discussing it). >>> >>> This is more opening the discussion and gathering feedback/feelings of >>> people. I think such decisions should be discussed (and their >>> consequences realized) well in-advance so that at the time of decision >>> we have all arguments and opinions and know the consequences. >>> >>> And it is not "strictly" necessary. It's just (like with all similar >>> cases) an increased burden for maintenance and extra/boilerplate code >>> that might be removed. In those cases there is rarely a "0/1" decision >>> that can be made "here we know for sure that we should increase >>> min-version" - it's more of the collective thinking: "Are the >>> burden/overhead heavy enough we don't want to carry it any more as a >>> community". >>> >>> J.
