+1 >From my side the proposal handles all concerns I raised in previous threads. I think mixed-governance is a step in the right direction.
On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > This is a follow-up after a few discussions started about providers that > were put on hold around the summit. I held a number of discussions during > theSummit and after, and as result I think I have a proposal that can move > forward some of the "stalled" decisions we need to make. > > *TL;DR;* > > My proposal is a "mixed-governance" model where "stakeholders" are more > responsible for cherry-picking and testing their providers (including > system testing) while Airflow PMC members will continue to be responsible > for releasing them. > > *Why do we need that?* > > Google, Amazon and possibly others teams who are interested in maintaining > more backwards compatible versions of their providers will commit to make > PRs of the cherry-picks for older release branches of their providers. > Those providers we release in parallel with the latest versions during the > normal provider cycle. We can deprecate changes more aggressively in the > "latest" release if we do that. > > Those cherry-picked PRs will be driven, tested and performed by the > stakeholder teams (Google/Amazon, Databricks, others) and will only contain > cherry-picks, while we - as PMC - will release them following the ASF rules > (this is very important for the ASF to follow strict release policies > regarding who and how performs releases). > > This also allows us to introduce similar rules for new provider's > acceptance for new providers for "main releases". It also allows running > the "system tests" for the provider under control of the stakeholder (after > applying AIP-47 changes). > > *Example 1*: Google team can cherry-pick changes to a google-provider-6 > branch and then we release a google 6.8.1 or 6.9.0 provider with some of > the bug-fixes and features (together with - say latest 8.1.0). > > *Example 2*: DataLake provider from Databricks - can get accepted if > Databricks commits to maintaining it. We will release the provider as long > as Databricks maintains it. > > *Longer context:* > > I have - in my mind so far - a longer roadmap for providers that will lead > them to be separated from the core and I want to write an AIP about that > soon. This AIP will detail all the steps needed - I will work with multiple > interested parties on it and it will take some time to agree and complete. > But I want to start with something tangible that will solve quite a few > problems that were raised recently and something that seems to be possible > to be solved in the current provider release cycle (till the end of June) > and test some of the governance approach. > > This proposal simply builds on our semver approach - we do not change it, > we just start releasing some providers (those that have some backing from > stakeholders) in more than one version - including "latest" and earlier, > more backwards-compatible branches. Not all providers - just some. Not all > branches - just those that the stakeholders will commit to maintain. > > We need such a commitment from stakeholders, because we - as the > Airflow community and maintainers, want to only actively maintain the > latest releases, where it is in the interest of the stakeholders to > cherry-pick and test also earlier, more backwards compatible releases of > their choice. > > *What problems this proposal solves:* > > ** Problem 1*: DEPRECATION REMOVAL > > we can remove deprecations faster in "main" versions of the providers - no > need to introduce a deprecation policy - the stakeholders for the providers > will take care about cherry-picking and maintaining more > "backwards-compatible" versions. We are free to remove deprecations in > major releases (in a cherry-pickable way of course). > > ** Problem 2*: PROVIDERS DIVERGENCE > > We avoid the problem (already happened with the composer release) that the > stakeholders in a given provider had to release their own version which was > not available in their community - with some cherry-picks. We want to avoid > "diverging" there - by releasing the cherry-picked providers by the > community, we also give other users an opportunity to follow "slower" > deprecation policies for as long as it is maintained. > > ** Problem 3*: PROVIDERS GOVERNANCE MODEL > > We are going to test a governance model that we might apply when we split > providers. We are talking about it for quite some time - but this is what > helps us to test the model where stakeholders provide more "maintenance" > while the community still takes care about releases. We (as community) can > commit to releasing such a version of a provider as long as the stakeholder > will actively maintain it. We can stop at any moment if we do not have > support from the stakeholder. If it works - we can keep it as a long-term > solution. In the future we can think of other scenarios (passing ownership > of a provider to stakeholders who want it - providing we want it too) but > we can decide about it when we learn from the mixed-governance model and > see if it works. > > ** Problem 4*: ACCEPTING NEW PROVIDERS > > If this is an acceptable approach - we can also apply a very > similar governance model to adding new providers and that should unblock > some of the PRs that are waiting for our decision. Knowing that we are > going to split and that we can expect "commitment" from a stakeholder, we > should be able to accept new providers. This might be possible assuming > that the stakeholder will make a similar commitment - but for new > providers, that commitment might also have to cover reviewing and testing > new changes. We might also decide as a community to stop releasing new > providers there if such support is missing. This way we can set the > expectations we have as a community for new providers - we will release > them as long as the stakeholder will actively make sure it is maintained. > > ** Problem 5.* SPLITTING PROVIDERS FROM CORE > > We all know we want to split providers from core. By introducing > mixed-governance we can test if it will work for the providers before we > split them. It will take some time (and detailed AIP) to split, but in the > meantime we can see if we will be able to apply the mixed-governance after > the split. We will see if we can agree when it comes to expectations and > find solutions before we actually split. > > J. >
