+1 -- We have discussed this during the Airflow Summit in-person with Ash, Rafal (and his team), Jarek and I about this for a long time, and I think this is a good step forward.
Regards, Kaxil On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]> wrote: > I think we should continue to be strictly vendor-neutral. No organization > should be able to gain special privileges or control a project’s direction. > > pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]> napisał(a): > >> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >> committers, because only they have write permission to the apache/airflow >> repository. As far as I know, Github does not allow us to grant write-only >> permissions to the selected branch. >> >> We can keep these branches in forks managed by stakeholders teams, but I >> am afraid of the benefit that it will be then copied by us to our >> repository and then released by us. If the release was prepared by an >> external team, I think we should make it clear that it was prepared by >> another team, including by publishing on the Pypi account of the team that >> dealt with it. >> >> I think that everything Apache PMC releases should be prepared and >> created fully within the apache / airflow repositories. If stakeholders >> team do not have such a possibility, we should figure out that these teams >> become part of the community, and therefore work together with the entire >> community, not in isolation. Only then will we be able to act in accordance >> with the Apache Way <https://www.apache.org/theapacheway/>, in >> particular each individual person will be able to contribute to the >> community as an individual, and not as a company or stakeholders team >> (Community of Peers) and no person will get special privileges just on the >> basis of their employment status (Earned Authority) >> >> >> >> pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]> napisał(a): >> >>> > Will the people who maintain the providers' packages have the commiter >>> >>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>> cherry-picking can be done in separate branches (and in this case in >>> forks). >>> Then the branch can be fast-forwarded by the committer in the "airflow" >>> repo. No problem with that. >>> >>> J. >>> >>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]> >>> wrote: >>> >>>> Will the people who maintain the providers' packages have the commiter >>>> status? I guess it is necessary for people to have write access to the >>>> repository and therefore to be able to make cherry-pick changes to the >>>> branch. >>>> >>>> pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]> napisał(a): >>>> > >>>> > +1 >>>> > From my side the proposal handles all concerns I raised in previous >>>> threads. >>>> > I think mixed-governance is a step in the right direction. >>>> > >>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> >>>> wrote: >>>> >> >>>> >> Hello everyone, >>>> >> >>>> >> This is a follow-up after a few discussions started about providers >>>> that were put on hold around the summit. I held a number of discussions >>>> during theSummit and after, and as result I think I have a proposal that >>>> can move forward some of the "stalled" decisions we need to make. >>>> >> >>>> >> TL;DR; >>>> >> >>>> >> My proposal is a "mixed-governance" model where "stakeholders" are >>>> more responsible for cherry-picking and testing their providers (including >>>> system testing) while Airflow PMC members will continue to be responsible >>>> for releasing them. >>>> >> >>>> >> Why do we need that? >>>> >> >>>> >> Google, Amazon and possibly others teams who are interested in >>>> maintaining more backwards compatible versions of their providers will >>>> commit to make PRs of the cherry-picks for older release branches of their >>>> providers. Those providers we release in parallel with the latest versions >>>> during the normal provider cycle. We can deprecate changes more >>>> aggressively in the "latest" release if we do that. >>>> >> >>>> >> Those cherry-picked PRs will be driven, tested and performed by the >>>> stakeholder teams (Google/Amazon, Databricks, others) and will only contain >>>> cherry-picks, while we - as PMC - will release them following the ASF rules >>>> (this is very important for the ASF to follow strict release policies >>>> regarding who and how performs releases). >>>> >> >>>> >> This also allows us to introduce similar rules for new provider's >>>> acceptance for new providers for "main releases". It also allows running >>>> the "system tests" for the provider under control of the stakeholder (after >>>> applying AIP-47 changes). >>>> >> >>>> >> Example 1: Google team can cherry-pick changes to a >>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>> provider with some of the bug-fixes and features (together with - say >>>> latest 8.1.0). >>>> >> >>>> >> Example 2: DataLake provider from Databricks - can get accepted if >>>> Databricks commits to maintaining it. We will release the provider as long >>>> as Databricks maintains it. >>>> >> >>>> >> Longer context: >>>> >> >>>> >> I have - in my mind so far - a longer roadmap for providers that >>>> will lead them to be separated from the core and I want to write an AIP >>>> about that soon. This AIP will detail all the steps needed - I will work >>>> with multiple interested parties on it and it will take some time to agree >>>> and complete. But I want to start with something tangible that will solve >>>> quite a few problems that were raised recently and something that seems to >>>> be possible to be solved in the current provider release cycle (till the >>>> end of June) and test some of the governance approach. >>>> >> >>>> >> This proposal simply builds on our semver approach - we do not >>>> change it, we just start releasing some providers (those that have some >>>> backing from stakeholders) in more than one version - including "latest" >>>> and earlier, more backwards-compatible branches. Not all providers - just >>>> some. Not all branches - just those that the stakeholders will commit to >>>> maintain. >>>> >> >>>> >> We need such a commitment from stakeholders, because we - as the >>>> Airflow community and maintainers, want to only actively maintain the >>>> latest releases, where it is in the interest of the stakeholders to >>>> cherry-pick and test also earlier, more backwards compatible releases of >>>> their choice. >>>> >> >>>> >> What problems this proposal solves: >>>> >> >>>> >> * Problem 1: DEPRECATION REMOVAL >>>> >> >>>> >> we can remove deprecations faster in "main" versions of the >>>> providers - no need to introduce a deprecation policy - the stakeholders >>>> for the providers will take care about cherry-picking and maintaining more >>>> "backwards-compatible" versions. We are free to remove deprecations in >>>> major releases (in a cherry-pickable way of course). >>>> >> >>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>> >> >>>> >> We avoid the problem (already happened with the composer release) >>>> that the stakeholders in a given provider had to release their own version >>>> which was not available in their community - with some cherry-picks. We >>>> want to avoid "diverging" there - by releasing the cherry-picked providers >>>> by the community, we also give other users an opportunity to follow >>>> "slower" deprecation policies for as long as it is maintained. >>>> >> >>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>> >> >>>> >> We are going to test a governance model that we might apply when we >>>> split providers. We are talking about it for quite some time - but this is >>>> what helps us to test the model where stakeholders provide more >>>> "maintenance" while the community still takes care about releases. We (as >>>> community) can commit to releasing such a version of a provider as long as >>>> the stakeholder will actively maintain it. We can stop at any moment if we >>>> do not have support from the stakeholder. If it works - we can keep it as a >>>> long-term solution. In the future we can think of other scenarios (passing >>>> ownership of a provider to stakeholders who want it - providing we want it >>>> too) but we can decide about it when we learn from the mixed-governance >>>> model and see if it works. >>>> >> >>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>> >> >>>> >> If this is an acceptable approach - we can also apply a very similar >>>> governance model to adding new providers and that should unblock some of >>>> the PRs that are waiting for our decision. Knowing that we are going to >>>> split and that we can expect "commitment" from a stakeholder, we should be >>>> able to accept new providers. This might be possible assuming that the >>>> stakeholder will make a similar commitment - but for new providers, that >>>> commitment might also have to cover reviewing and testing new changes. We >>>> might also decide as a community to stop releasing new providers there if >>>> such support is missing. This way we can set the expectations we have as a >>>> community for new providers - we will release them as long as the >>>> stakeholder will actively make sure it is maintained. >>>> >> >>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>> >> >>>> >> We all know we want to split providers from core. By introducing >>>> mixed-governance we can test if it will work for the providers before we >>>> split them. It will take some time (and detailed AIP) to split, but in the >>>> meantime we can see if we will be able to apply the mixed-governance after >>>> the split. We will see if we can agree when it comes to expectations and >>>> find solutions before we actually split. >>>> >> >>>> >> J. >>>> >>>
