Hi, Talking on behalf of all users of the Airflow Google provider package... we would like to exclude apache-airflow-providers-google from the initiative of removing all of the deprecated operators at once. Instead, we would like to respect the 6-month-long deprecation period and remove the deprecated operators incrementally.
We will collaborate with Elad how to: a) make this process as optimal as possible b) Google team can be deeply involved, i.e. responsible for incremental removals of the operators c) We will collaborate with Elad and release managers to properly document the changes in apache-airflow-providers-google so users know which operators they need to switch to. d) we will try to evaluate if there are any operators that maybe are not used at all and maybe we could deprecate them faster than 6 months. We will also make it very explicit that the deprecated operators are not meant to be used with Airflow 3. Elad - pls, chime in if necessary. Regards, Rafal On Mon, Nov 18, 2024 at 6:52 AM Ephraim Anierobi <ephraimanier...@gmail.com> wrote: > +1, not just for removing all deprecations before 2.11 but to have a policy > of removing the deprecations on every major provider release. But then, we > have to be careful which version of Airflow these major providers' bumps > would be part of as constraints so that we don't include a major bump of > providers in the Airflow patch releases’ constraints file, as that would be > a breaking change. > > On Sun, 17 Nov 2024 at 12:41, Hussein Awala <huss...@awala.fr> wrote: > > > +1 for removing all deprecations from providers before Airflow 2.11. > > > > Unfortunately, we don't have a policy to remove providers' deprecations, > as > > we see in some providers we still maintain deprecated features after more > > than 4 major releases, this is something we have discussed multiple times > > in the past, and I think we should re-discuss it and take a decision > before > > releasing Airflow 3. I propose to remove deprecations after 2 major > > releases and to cover what Jarek mentioned about some of the providers > > managed by stakeholders, we can consider this as the default policy, > which > > we can replace with a provider-level policy. > > > > On Sun, Nov 17, 2024 at 11:56 AM Jens Scheffler > <j_scheff...@gmx.de.invalid > > > > > wrote: > > > > > Strong +1 > > > > > > Which means for almost all providers we should attempt to release a > > > "breaking version" and effectively remove deprecated functions. > > > > > > Just looking into two examples: > > > > > > - providers/src/airflow/providers/standard/operators/bash.py:170 - > > > "skip_exit_code is deprecated. Please use skip_on_exit_code" > > > > > > - providers/src/airflow/providers/docker/decorators/docker.py:130 - > > > "`use_dill` is deprecated and will be removed in a future version." > > > > > > ..and then airflow-providers-docker would be bumped to 4.0.0... > > > > > > Correct understood? > > > > > > Jens > > > > > > On 17.11.24 10:59, Jarek Potiuk wrote: > > > > +1 . Very good and well thought proposal coming from a lot of > > discussions > > > > we held on slack with a number of people. > > > > > > > > Great for the users and also recognizing our stakeholders needs and > > their > > > > willingness to maintain their providers in the way it is best for > their > > > > user base (both within their services and outside) > > > > > > > > While we know such an exclusion is now requested (and very consistent > > and > > > > good policy proposed) by Google (we should likely register the policy > > > > somewhere in Google provider's readme BTW) - similar ones.could be > > added > > > by > > > > other stakeholders not only as big as Google, Amazon and Microsoft. > > > > > > > > But also smaller providers could benefit from it if they had any > stake > > > > lholder who would like to define and follow through their deprecation > > > > policy - for example Teradata could do the same already but others > are > > > more > > > > than welcome to follow and both - set the rules and commit to > > maintaining > > > > them. This is a great opportunity for others to step and raise their > > hand > > > > and say 'hey I would like to take more responsibility for that > provider > > > and > > > > here are the rule I propose and commit to keep it maintained. > > > > > > > > BTW. I know Microsoft team (kudos to Ambika!) is working on system > test > > > > dashboard (yay!!!) so finally we may get have all the stakeholders > from > > > the > > > > big clouds involved and active in our community, and also having > > > Astronomer > > > > as leading 3.0 and of course biggest contributor, and all the > > individual > > > > contributors - that makes it a very nice, sustainable community > around > > > > Airflow and makes me really bullish on the future of Airflow! > > > > > > > > It's a really great to see how we all collaborate respect and > recognize > > > > each-others needs. > > > > > > > > J > > > > > > > > niedz., 17 lis 2024, 09:30 użytkownik Elad Kalif <elad...@apache.org > > > > > > napisał: > > > > > > > >> Hello everyone, > > > >> > > > >> We know from a previous survey that users tend to upgrade providers > > > >> together with Airflow core. This is also consistent with what I hear > > > from > > > >> talking to users in the community. Upgrading the provider separately > > is > > > >> done mostly when there is a need (relevant bug fix, needed feature, > > > etc...) > > > >> This is something that we should prepare for. While we treat > providers > > > as > > > >> independent packages users may see this as part of Airflow core. > > > >> > > > >> I can say for myself that one of the challenges of upgrading 1.10 > -> 2 > > > was > > > >> provider packages. > > > >> > > > >> My suggestion is to take the risk / problem away. If we remove all > > > >> deprecations users will have to handle them while they are on > Airflow > > > 2.x > > > >> and before they are migrating to Airflow 3.x - To my perspective > this > > > will > > > >> allow a smoother migration. > > > >> The reason I suggest to do it before Airflow 2.11 is because we > treat > > > 2.11 > > > >> as a bridge release thus I'd like the constraints for this release > to > > > have > > > >> the provider versions after removal of deprecations. > > > >> > > > >> This is a process we already started with cncf.kubernetes and amazon > > > >> providers that removed all their deprecations: > > > >> https://github.com/apache/airflow/pull/43689 > > > >> https://github.com/apache/airflow/pull/42450 > > > >> > > > >> *I'd like to focus the discussion around the following proposal:* > > > >> We will remove all deprecations from providers prior to Airflow 2.11 > > > >> (Providers that already did this recently don't have to do it again > if > > > we > > > >> added new deprecations), Given the mixed governance model for > > providers > > > ( > > > >> > > > >> > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#mixed-governance-model-for-3rd-party-related-community-providers > > > >> ) Stackholders who maintain a specific provider can ask to be > > excluded. > > > The > > > >> exclusion will be registered and referenced in the guides/tools we > > will > > > >> introduce for Airflow 3 migration. > > > >> > > > >> Regards, > > > >> Elad > > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
