Hello everyone, This is a follow-up after a few discussions started about providers that were put on hold around the summit. I held a number of discussions during theSummit and after, and as result I think I have a proposal that can move forward some of the "stalled" decisions we need to make.
*TL;DR;* My proposal is a "mixed-governance" model where "stakeholders" are more responsible for cherry-picking and testing their providers (including system testing) while Airflow PMC members will continue to be responsible for releasing them. *Why do we need that?* Google, Amazon and possibly others teams who are interested in maintaining more backwards compatible versions of their providers will commit to make PRs of the cherry-picks for older release branches of their providers. Those providers we release in parallel with the latest versions during the normal provider cycle. We can deprecate changes more aggressively in the "latest" release if we do that. Those cherry-picked PRs will be driven, tested and performed by the stakeholder teams (Google/Amazon, Databricks, others) and will only contain cherry-picks, while we - as PMC - will release them following the ASF rules (this is very important for the ASF to follow strict release policies regarding who and how performs releases). This also allows us to introduce similar rules for new provider's acceptance for new providers for "main releases". It also allows running the "system tests" for the provider under control of the stakeholder (after applying AIP-47 changes). *Example 1*: Google team can cherry-pick changes to a google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 provider with some of the bug-fixes and features (together with - say latest 8.1.0). *Example 2*: DataLake provider from Databricks - can get accepted if Databricks commits to maintaining it. We will release the provider as long as Databricks maintains it. *Longer context:* I have - in my mind so far - a longer roadmap for providers that will lead them to be separated from the core and I want to write an AIP about that soon. This AIP will detail all the steps needed - I will work with multiple interested parties on it and it will take some time to agree and complete. But I want to start with something tangible that will solve quite a few problems that were raised recently and something that seems to be possible to be solved in the current provider release cycle (till the end of June) and test some of the governance approach. This proposal simply builds on our semver approach - we do not change it, we just start releasing some providers (those that have some backing from stakeholders) in more than one version - including "latest" and earlier, more backwards-compatible branches. Not all providers - just some. Not all branches - just those that the stakeholders will commit to maintain. We need such a commitment from stakeholders, because we - as the Airflow community and maintainers, want to only actively maintain the latest releases, where it is in the interest of the stakeholders to cherry-pick and test also earlier, more backwards compatible releases of their choice. *What problems this proposal solves:* ** Problem 1*: DEPRECATION REMOVAL we can remove deprecations faster in "main" versions of the providers - no need to introduce a deprecation policy - the stakeholders for the providers will take care about cherry-picking and maintaining more "backwards-compatible" versions. We are free to remove deprecations in major releases (in a cherry-pickable way of course). ** Problem 2*: PROVIDERS DIVERGENCE We avoid the problem (already happened with the composer release) that the stakeholders in a given provider had to release their own version which was not available in their community - with some cherry-picks. We want to avoid "diverging" there - by releasing the cherry-picked providers by the community, we also give other users an opportunity to follow "slower" deprecation policies for as long as it is maintained. ** Problem 3*: PROVIDERS GOVERNANCE MODEL We are going to test a governance model that we might apply when we split providers. We are talking about it for quite some time - but this is what helps us to test the model where stakeholders provide more "maintenance" while the community still takes care about releases. We (as community) can commit to releasing such a version of a provider as long as the stakeholder will actively maintain it. We can stop at any moment if we do not have support from the stakeholder. If it works - we can keep it as a long-term solution. In the future we can think of other scenarios (passing ownership of a provider to stakeholders who want it - providing we want it too) but we can decide about it when we learn from the mixed-governance model and see if it works. ** Problem 4*: ACCEPTING NEW PROVIDERS If this is an acceptable approach - we can also apply a very similar governance model to adding new providers and that should unblock some of the PRs that are waiting for our decision. Knowing that we are going to split and that we can expect "commitment" from a stakeholder, we should be able to accept new providers. This might be possible assuming that the stakeholder will make a similar commitment - but for new providers, that commitment might also have to cover reviewing and testing new changes. We might also decide as a community to stop releasing new providers there if such support is missing. This way we can set the expectations we have as a community for new providers - we will release them as long as the stakeholder will actively make sure it is maintained. ** Problem 5.* SPLITTING PROVIDERS FROM CORE We all know we want to split providers from core. By introducing mixed-governance we can test if it will work for the providers before we split them. It will take some time (and detailed AIP) to split, but in the meantime we can see if we will be able to apply the mixed-governance after the split. We will see if we can agree when it comes to expectations and find solutions before we actually split. J.
