I think it's cool we are discussing it - this is the gist of Apache
Way and I think we have very clear rules here that not the strongest
opinion or longest email matter but consensus - if we can reach it,
and vote (if it seems we need to vote).
That's the power and strength of our community. And as usual - I would
love to hear other voices of the wider community - more opinions from
users, other committers and stakeholders.
Whether we like it or not - providers are part of our PMC and they
follow the same rules as the core Airflow code (at least until we
decide otherwise).
They are not a separate PMC. We have the same people and rules to
govern them. That's the current state at least. If we want to separate
them out to a different PMC and make them different "PMC" they would
have to go through an incubation process (I actually talked about it
recently with some ASF old-timers and this is what they told me at
least).
I hope the "strong" opinions here would not discourage those voices
:). I also think that people meeting in-person at the summit next week
is a great opportunity to discuss those topics related to providers. I
think we have at least a few things related to providers that we will
have to decide soon, so more voices and opinions there will - I hope
make it possible to make good decisions.
I think maybe a good time to summarise where we are so it will give
everyone looking here a chance to think that through. I'll try (I hope
:) objectively summarise where we are and what options we are
considering:
1) The community vs. 3rd-party providers:
- currently we do not have any rules and basically we decide in
PRs whether we accept a new provider or not - it's up to individual
decision of the individual committers
Options we have:
a) we keep the status quo and decide individually (in PRs) on
the spot for each provider
b) we might want to set some rules based on some criteria
(for example popularity/availability of stakeholder to manage
3rd-party provider etc.) and start following it
c) we might want to stop accepting any providers by default
and only add one when we explicitly agree to it (by voting i guess)
2) The min airflow version in providers:
- currently the decision is that the next release of providers
will bump min version of Airflow to 2.2
Options we have:
a) we keep the status quo and release new providers with min-version 2.2
b) we change the policy (somebody has to propose it and
convince others it's better) to do it only when needed by individual
providers
3) Removal policy for deprecated features in providers.
- currently we have no special rules. SemVer does not mandate the
removal so effectively the rule is "we MAY remove it but we do not
have to". This is basically what we've been doing so far - individual
contributors proposed removals and they were individually accepted by
the committers - they decided on the spot. Any breaking change in the
provider resulted in bumping the major version
Options we have:
a) we keep the status quo and continue leaving the decision by
the committers on-the-spot
b) we mandate removal of all deprecations at the major bump
after the deprecation was introduced
c) we mandate (or make possible)removal of all deprecations
together with bumping min-version of the provider (my proposal)
d) we mandate {or make possible) remove of all deprecations at
the second major bump after the deprecation was introduced
e) we mandate (or make possible) removal of all deprecation
ain major bump when some time passed (3/6 months were mentioned)
Just to explain - I believe options c) - e) have two variations: we
can either mandate removal of deprecations or simply make it
possible. Option b) has only "mandate" as "make possible" is the same
as a) in this case - we may or may not remove deprecation with major
bump only which is a strict SemVer requirement.
BTW, IMHO - mandating has the consequence that whenever PR is made
with breaking change, it should also - by definition - remove all
deprecations, so if we accept any breaking change in a provider, it
should be connected with removal of all deprecations. It should really
be done with the first breaking change, because otherwise we are not
able to "make sure" it happens - we have no-one to walk through all
providers and remove all deprecations at release time, so all
deprecations have to be removed in the same PR when the first breaking
change is added - we should be aware about this consequence.
J.
On Sun, May 22, 2022 at 12:07 AM Kaxil Naik <[email protected]> wrote:
>>
>> It just means that ANY major release can remove deprecated code regardless
>> of when we introduced the deprecation.
>
>
> Precisely, yes.
>
> On Sat, 21 May 2022 at 20:23, Elad Kalif <[email protected]> wrote:
>>
>> rethinking about this I do support Ash's approach.
>>
>> > Following the above. Are you proposing we make it as a rule?
>>
>> The rule is that we can. It doesn't mean that we will every time.
>> It just means that ANY major release can remove deprecated code regardless
>> of when we introduced the deprecation.
>> The community is welcome to express their opinions on specific cases -
>> everything is transparent.
>>
>> > If users were caught by surprise with breaking changes in major versions,
>> > then maybe the other major versions we released were "not breaking".
>>
>> The surprise I'm referring to is that this is the first time we actually
>> removed deprecated features in providers. When I removed deprecated features
>> for Google 7.0.0 release I didn't expect it to be a surprise. I only
>> learned it after the release which is why I started this thread.
>>
>> On Sat, May 21, 2022 at 10:04 PM Kaxil Naik <[email protected]> wrote:
>>>
>>> I don't think one size fits all approach will work with Providers. The wait
>>> for X months without a major version won't be possible to achieve in cases
>>> where we need to bump dependencies of providers that break things too.
>>> (example Google provider 3.0.0 -> 4.0.0)
>>>
>>> If users were caught by surprise with breaking changes in major versions,
>>> then maybe the other major versions we released were "not breaking".
>>>
>>> Instead of adding tons of policies, sticking by SemVer and bumping major
>>> versions only when we have breaking changes for providers is what I
>>> recommend as well.
>>>
>>> In Rafal's case, this was the issue because the fixes that were clubbed
>>> with Google Provider might have been critical or major. I just say in cases
>>> like those we assess and help the community by releasing a patch version of
>>> the previous release by assessing if that's possible and for stakeholders
>>> to just look at the changes when we are in the voting phase. This should be
>>> a comparatively rare occurrence.
>>>
>>> Regards,
>>> Kaxil
>>>
>>>
>>> On Sat, 21 May 2022 at 19:24, Jarek Potiuk <[email protected]> wrote:
>>>>
>>>> And about this
>>>>
>>>> > By "policy for policy's sake" this is exactly what I mean - we have so
>>>> > much work down already it feels nigh on impossible to know everything.
>>>>
>>>> BTW. I see it differently. We have policies, precisely because we are
>>>> not able to follow everything, so we write policies that anyone who is
>>>> a committer can follow - without getting approvals or discussing it
>>>> with others. We should have those policies done in the way that anyone
>>>> who is in the project knows what to do without making any assumptions
>>>> :). So if we feel overwhelmed with decisions - we should have more
>>>> precise policies so that - we do not have to make the decisions next
>>>> time when we get there. That's why I think we should codify all the
>>>> rules we follow - especially when this impacts our users.
>>>>
>>>> And I think it is super important that our users know what to expect from
>>>> us :).
>>>>
>>>> > But it's a sign to the users that something has broken. If you want to
>>>> > wait six months feel free. But I don't think that should be our policy.
>>>> > I the policy to be "we release under SemVer so can delete code that was
>>>> > deprecated under the previous major version". No time requirement.
>>>>
>>>> Following the above. Are you proposing we make it as a rule? Always
>>>> remove all deprecation with any breaking release of every provider? I
>>>> think our users should know what to expect.
>>>>
>>>> BTW. Looking at the users commenting above - this is far too
>>>> aggressive for them and I agree it is.
>>>>
>>>> J.
>>>>
>>>>
>>>> J.
>>>>
>>>> On Sat, May 21, 2022 at 8:20 PM Ash Berlin-Taylor <[email protected]> wrote:
>>>> >
>>>> > https://github.com/semver/semver/blob/master/semver.md.
>>>> >
>>>> > Nothing says when you have to make breeding changes of course.
>>>> >
>>>> > But it's a sign to the users that something has broken. If you want to
>>>> > wait six months feel free. But I don't think that should be our policy.
>>>> > I the policy to be "we release under SemVer so can delete code that was
>>>> > deprecated under the previous major version". No time requirement.
>>>> >
>>>> > On 21 May 2022 19:12:32 BST, Jarek Potiuk <[email protected]> wrote:
>>>> >>>
>>>> >>> Will min Airflow version aside: we agreed we're going to follow
>>>> >>> SemVer, so I say we stick to that. X+1.0.0 (when ever we choose to
>>>> >>> release it) is when we remove the deprecated code.
>>>> >>
>>>> >>
>>>> >> That's not entirely correct. Unfortunately semver.org is down now so I
>>>> >> cannot check my authoritative source :). But - from what I know,
>>>> >> SemVer does not really mandate removal of deprecations.
>>>> >> It says that (from memory) MAJOR version MAY contain breaking
>>>> >> changes, but it never (unless I am mistaken) states that you MUST make
>>>> >> braking changes for all deprecations.
