One more thing. This one is essentially impossible (Hyrum's law explains that very well https://www.hyrumslaw.com/:
> The change is considered to be a breaking (in the context of google providers package) if a DAG that was working before, stops to work after the change. I propose to change it to: > The change is considered to be a breaking if a DAG using the providers (in the way they were intended to be used), that was working before, stops to work after the change. It is absolutely not possible to make sure that every single DAG written by someone will continue working after any change. Literally. For example if someone extends BigQueryOperator and calls (self._this_very_much_internal_method()` in his operator's new execute(), and we rename the method, their DAG will stop working. Is it breaking ? No. Is the user using it with the intention how it should be used? Absolutely not. SemVer - is not a promise things won't break, it's a promise that our intention is that things won't break. But whether the way our users use it will make it break or not, that's another story. J On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> wrote: > BTW. Another example of an exception would be a security fix. We **migh** > find a security issue that will require a breaking change. But then again - > the breaking change might **only** cover that security fix - we do not have > to remove all the deprecations in this case (but we could if we decide so - > very much depending on nature of those deprecations) > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> I think it's a good idea to codify the rules, indeed what Elad mentioned, >> we **mostly** follow a very similar approach, but it's not written >> anywhere. >> >> >> Re: points of Elad: >> >> 1. I am ok with 6 months, but I agree here we should not have "strict" >> limits. It's fine to have some default (i.e. accumulate deprecating changes >> for some time - max. 6 months) - but I think the policy should have an >> option to apply exceptions - for example when we have a major change in >> many libraries and we need to make some breaking changes (ads case) we >> shall do it (but in this case we don't have to immediately remove all >> deprecations - they can wait for the regular (say 6 months) breaking change >> that will be coming. The users can - if needed - even in new versions of >> Airflow - downgrade to previous provider versions in case they have problem >> with it. >> >> I like the idea of having some time (6 months) where we can very safely >> say "OK, enough is enough, lets remove deprecations" - and this proposal >> addresses it nicely. Currently it's purely subjective judgment. One small >> thing here to clarify for "accumulating" deprecations - I think we should >> make sure the deprecation is present in already released minor/patchlevel >> version to be able to remove it (so even if it is deprecated 2 months ago >> but we had release with the deprecation, it's ok to remove it in the next >> breaking change. >> >> 2. Agree (and again here "strict" is not good I think). I'd treat that 6 >> months as a baseline. Before that - we should have very good reason to make >> a breaking change (and we might decide to do it partially only) - after - >> we are free to remove things that have been already announced as deprecated >> and we should consider making breaking change whenever we release a new >> version for whatever reason (but again - not mandatory). I's just an >> indication that tells us "Last breaking release was 6 months ago, it's now >> OK to remove all the deprecations without having an extremely good reason". >> >> Very much agree with 3. -> this should be the same as for other >> providers. We should adopt the same rule for all of them. And if we apply >> the adjustments above - we should - I think - be ok to apply it to any >> provider I think. >> >> J >> >> >> >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <elad...@apache.org> wrote: >> >>> This is not too much different than what we already do. >>> We deprecate then we remove. We almost never create breaking change >>> without >>> deprecating first. >>> >>> I'd like to raise a few notes: >>> >>> 1. Major releases can happen frequently. I see no problem with it. This >>> is >>> more a question of what are the changes. If google has 3 major releases >>> one >>> with breaking change to ads, another to leveldb and 3rd to cloud these >>> are >>> not related one to another and unlikely that there is 1 user that uses >>> all >>> of these together. We create major releases when needed and we keep a >>> close >>> eye and challenge why a change must be a breaking change. If possible we >>> will always prefer to be backward compatible. >>> 2. Time based deprecations are not a good approach. Some changes require >>> much more time to be acted upon and others may be trivial. >>> As a rule we almost never cut a breaking change release just to remove >>> deprecations. >>> 3. Setting a policy per provider is not the right way to go. We have a >>> shared governance model for provider handling and I as release manager >>> will >>> do what I can to accommodate requests from companies who participate in >>> managing the provider. Which means that if Google wants to remove >>> deprecations and ask for a breaking change release most likely we will >>> accommodate their request. My point of view is that the company knows >>> what >>> is best for their own users. I don't think that what is right for Google >>> must bind AWS or Microsoft. 1 policy goes against the shared >>> governance idea. >>> >>> >>> >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> >>> wrote: >>> >>> > Hi. >>> > >>> > I would like to discuss/propose a deprecation policy for >>> > apache-airflow-providers-google package, mostly for deprecating Airflow >>> > operators (but not limited to it). >>> > >>> > *Some background:* >>> > Airflow google provider package (as most of other provider packages in >>> > Airflow) has a tendency to constantly evolve. >>> > There are cases when its features or operators/hooks have to be >>> deprecated. >>> > The typical example of such a case is “refactoring” which leads to the >>> > renaming of operators, parameters or public methods. Such refactorings >>> > maybe consequence of: >>> > - using a new dependent library (that causes some methods to be >>> deprecated) >>> > - using a new version of API (that causes some parameters to be >>> deprecated) >>> > - restructuring/migration of the operators (that causes some operators >>> to >>> > be deprecated) >>> > Given the need of a “deprecation” procedure, it looks essential to >>> > establish policy around it to have users of Airflow google provider >>> package >>> > clearly understand when and how to adapt their DAGs. >>> > >>> > *Preambula (versioning of the package):* >>> > As mentioned in “Airflow’s release process and version policy” ( >>> > >>> > >>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy >>> > ) >>> > google provider package (and others) should follow SemVer, meaning >>> that any >>> > breaking changes should be released together with bumping major >>> version of >>> > the package. >>> > The change is considered to be a breaking (in the context of google >>> > providers package) if a DAG that was working before, stops to work >>> after >>> > the change. >>> > >>> > *My proposal (for discussion):* >>> > The entire procedure of deprecating (either method, parameter or >>> operator) >>> > consists of two steps: >>> > - Emission of the deprecation warning message >>> > Example of the message: >>> > “”” >>> > The “given” parameter is deprecated and will be removed after >>> dd.mm.yyyy. >>> > Please use “this” parameter instead. >>> > “”” >>> > - Once the date of the deprecated method/parameter/operator is passed, >>> it >>> > can be removed (together with bumping major version of the package) >>> > >>> > The format of the warning message may vary, but it has to contain: >>> > - What should be used instead >>> > - The date after which the method/parameter/operator will be removed >>> > >>> > *Additional considerations:* >>> > - Bumping a major version of the Airflow google provider package >>> shouldn’t >>> > happen very frequently (presumably not more frequently than 6 months), >>> > therefore when it happens it may accumulate removals of something that >>> was >>> > communicated to be removed even a couple of months ago. (related >>> discussion >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst) >>> > Example: if today is 2025-12-20 and we are releasing new major version >>> of >>> > the package, we can/should remove all deprecated >>> > methods/parameters/operators which have date prior to 2025-12-20 - it >>> can >>> > be November 2025, October 2025 or even earlier, since previous bump of >>> > major version happened e.g. in summer of 2025. >>> > - By default all deprecations should allow a 6 months time period until >>> > they will be removed and not available. This period will give Airflow >>> users >>> > enough time and flexibility to update their DAGs before actual removal >>> > happens. >>> > On a case by case basis this period can be adjusted given specific >>> > circumstances (e.g. in case deprecation is because of underlying API >>> sunset >>> > which can happen earlier than in 6 months). >>> > >>> > Apart from deprecation warning messages that will be printed as logs, >>> this >>> > information will be put as well in: >>> > - Airflow documentation >>> > - Release notes of Airflow google provider package >>> > >>> > Please, let me know what you think about this. >>> > >>> > -- >>> > Eugene >>> > >>> >>
