+1 I agree this is a logical next step towards possibly separating provider code from the Airflow code base (and it's useful even if we never do that).
Cheers, Niko ________________________________ From: Kamil Breguła <[email protected]> Sent: Monday, June 20, 2022 1:30 PM To: [email protected] Subject: RE: [EXTERNAL][PROPOSAL] Provider's mixed governance model - first step of provider separation CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. I discussed this problem with Jarek. The group of stakeholders will be open and everyone can join, not just Google employees, etc. Release branches will be maintained in apache/airflow repository. Any non-committer change will still require PR. This means there is no vendor neutrality risk. +1 I think this is a good step forward. pon., 20 cze 2022 o 21:18 Jarek Potiuk <[email protected]<mailto:[email protected]>> napisał(a): BTW. It will also be possible for anyone in the community to cherry-pick changes from main and make a PR (which also a committer will have to approve and merge). This is really no different that we have already done with cherry-picked commits to "v1-10-stable" and "v-2-3-stable" branches by non-committers. Random example here:https://github.com/apache/airflow/pull/14090 We do not give any privileges to the organisations. Quite the opposite - we make them responsible for preparing the PRs to be reviewed by committers. J. On Mon, Jun 20, 2022 at 9:07 PM Jarek Potiuk <[email protected]<mailto:[email protected]>> wrote: > I think we should continue to be strictly vendor-neutral. No organization > should be able to gain special privileges or control a project’s direction. This is strictly vendor-neutral - Kamil - we are going to release the same changes that we are releasing already in main providers, just selectively cherry-picked (and then reviewed and merged by committer to the branch in airflow repo) - why do you think it is non-vendor neutral? On Mon, Jun 20, 2022 at 9:04 PM Jarek Potiuk <[email protected]<mailto:[email protected]>> wrote: > We can keep these branches in forks managed by stakeholders teams, but I am > afraid of the benefit that it will be then copied by us to our repository and > then released by us. If the release was prepared by an external team, I think > we should make it clear that it was prepared by another team, including by > publishing on the Pypi account of the team that dealt with it. > Yes this is exactly what I proposed. Correction. I misread it. We are going to merge - only the cherry-picked changes that have been reviewed and merged by the committer. Same way as today. Just the process of cherry-picks is going to be done by the stakeholders, selecting the things to cherry-pick (all the changes to cherry-pick should be already merged in main). What we are going to do is to release subset of the changes we already approved (and released - because we are going to release those changes in the latest provider). So there will be no "new" changes in those forks - those will be just cherry-picked changes reviewed by the committer. There is no reason to mark it as "other" code - it will be the same changes that are going to be released anyway (just a subset of those). J. On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <[email protected]<mailto:[email protected]>> wrote: > Cherry-picking to branch v2-2* or 1.10.* can only be done by the committers, > because only they have write permission to the apache/airflow repository. As > far as I know, Github does not allow us to grant write-only permissions to > the selected branch. Kamil - you misunderstood it. The branch will be in the FORK of those users's choice - not in airflow repo. committer will merge that branch in the same way we do as today - but with fast-forwarding rather than squashing. > We can keep these branches in forks managed by stakeholders teams, but I am > afraid of the benefit that it will be then copied by us to our repository and > then released by us. If the release was prepared by an external team, I think > we should make it clear that it was prepared by another team, including by > publishing on the Pypi account of the team that dealt with it. Yes this is exactly what I proposed. > I think that everything Apache PMC releases should be prepared and created > fully within the apache / airflow repositories. If stakeholders team do not > have such a possibility, we should figure out that these teams become part of > the community, and therefore work together with the entire community, not in > isolation. Only then will we be able to act in accordance with the Apache > Way<https://www.apache.org/theapacheway/>, in particular each individual > person will be able to contribute to the community as an individual, and not > as a company or stakeholders team (Community of Peers) and no person will get > special privileges just on the basis of their employment status (Earned > Authority) Very true. And exactly follows my proposal :) J. On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <[email protected]<mailto:[email protected]>> wrote: +1 -- We have discussed this during the Airflow Summit in-person with Ash, Rafal (and his team), Jarek and I about this for a long time, and I think this is a good step forward. Regards, Kaxil On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]<mailto:[email protected]>> wrote: I think we should continue to be strictly vendor-neutral. No organization should be able to gain special privileges or control a project’s direction. pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]<mailto:[email protected]>> napisał(a): Cherry-picking to branch v2-2* or 1.10.* can only be done by the committers, because only they have write permission to the apache/airflow repository. As far as I know, Github does not allow us to grant write-only permissions to the selected branch. We can keep these branches in forks managed by stakeholders teams, but I am afraid of the benefit that it will be then copied by us to our repository and then released by us. If the release was prepared by an external team, I think we should make it clear that it was prepared by another team, including by publishing on the Pypi account of the team that dealt with it. I think that everything Apache PMC releases should be prepared and created fully within the apache / airflow repositories. If stakeholders team do not have such a possibility, we should figure out that these teams become part of the community, and therefore work together with the entire community, not in isolation. Only then will we be able to act in accordance with the Apache Way<https://www.apache.org/theapacheway/>, in particular each individual person will be able to contribute to the community as an individual, and not as a company or stakeholders team (Community of Peers) and no person will get special privileges just on the basis of their employment status (Earned Authority) pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]<mailto:[email protected]>> napisał(a): > Will the people who maintain the providers' packages have the commiter Nope - similarly as we do in v2-2* or what we did in 1.10.* cherry-picking can be done in separate branches (and in this case in forks). Then the branch can be fast-forwarded by the committer in the "airflow" repo. No problem with that. J. On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]<mailto:[email protected]>> wrote: Will the people who maintain the providers' packages have the commiter status? I guess it is necessary for people to have write access to the repository and therefore to be able to make cherry-pick changes to the branch. pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]<mailto:[email protected]>> napisał(a): > > +1 > From my side the proposal handles all concerns I raised in previous threads. > I think mixed-governance is a step in the right direction. > > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk > <[email protected]<mailto:[email protected]>> wrote: >> >> Hello everyone, >> >> This is a follow-up after a few discussions started about providers that >> were put on hold around the summit. I held a number of discussions during >> theSummit and after, and as result I think I have a proposal that can move >> forward some of the "stalled" decisions we need to make. >> >> TL;DR; >> >> My proposal is a "mixed-governance" model where "stakeholders" are more >> responsible for cherry-picking and testing their providers (including system >> testing) while Airflow PMC members will continue to be responsible for >> releasing them. >> >> Why do we need that? >> >> Google, Amazon and possibly others teams who are interested in maintaining >> more backwards compatible versions of their providers will commit to make >> PRs of the cherry-picks for older release branches of their providers. Those >> providers we release in parallel with the latest versions during the normal >> provider cycle. We can deprecate changes more aggressively in the "latest" >> release if we do that. >> >> Those cherry-picked PRs will be driven, tested and performed by the >> stakeholder teams (Google/Amazon, Databricks, others) and will only contain >> cherry-picks, while we - as PMC - will release them following the ASF rules >> (this is very important for the ASF to follow strict release policies >> regarding who and how performs releases). >> >> This also allows us to introduce similar rules for new provider's acceptance >> for new providers for "main releases". It also allows running the "system >> tests" for the provider under control of the stakeholder (after applying >> AIP-47 changes). >> >> Example 1: Google team can cherry-pick changes to a google-provider-6 branch >> and then we release a google 6.8.1 or 6.9.0 provider with some of the >> bug-fixes and features (together with - say latest 8.1.0). >> >> Example 2: DataLake provider from Databricks - can get accepted if >> Databricks commits to maintaining it. We will release the provider as long >> as Databricks maintains it. >> >> Longer context: >> >> I have - in my mind so far - a longer roadmap for providers that will lead >> them to be separated from the core and I want to write an AIP about that >> soon. This AIP will detail all the steps needed - I will work with multiple >> interested parties on it and it will take some time to agree and complete. >> But I want to start with something tangible that will solve quite a few >> problems that were raised recently and something that seems to be possible >> to be solved in the current provider release cycle (till the end of June) >> and test some of the governance approach. >> >> This proposal simply builds on our semver approach - we do not change it, we >> just start releasing some providers (those that have some backing from >> stakeholders) in more than one version - including "latest" and earlier, >> more backwards-compatible branches. Not all providers - just some. Not all >> branches - just those that the stakeholders will commit to maintain. >> >> We need such a commitment from stakeholders, because we - as the Airflow >> community and maintainers, want to only actively maintain the latest >> releases, where it is in the interest of the stakeholders to cherry-pick and >> test also earlier, more backwards compatible releases of their choice. >> >> What problems this proposal solves: >> >> * Problem 1: DEPRECATION REMOVAL >> >> we can remove deprecations faster in "main" versions of the providers - no >> need to introduce a deprecation policy - the stakeholders for the providers >> will take care about cherry-picking and maintaining more >> "backwards-compatible" versions. We are free to remove deprecations in major >> releases (in a cherry-pickable way of course). >> >> * Problem 2: PROVIDERS DIVERGENCE >> >> We avoid the problem (already happened with the composer release) that the >> stakeholders in a given provider had to release their own version which was >> not available in their community - with some cherry-picks. We want to avoid >> "diverging" there - by releasing the cherry-picked providers by the >> community, we also give other users an opportunity to follow "slower" >> deprecation policies for as long as it is maintained. >> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >> >> We are going to test a governance model that we might apply when we split >> providers. We are talking about it for quite some time - but this is what >> helps us to test the model where stakeholders provide more "maintenance" >> while the community still takes care about releases. We (as community) can >> commit to releasing such a version of a provider as long as the stakeholder >> will actively maintain it. We can stop at any moment if we do not have >> support from the stakeholder. If it works - we can keep it as a long-term >> solution. In the future we can think of other scenarios (passing ownership >> of a provider to stakeholders who want it - providing we want it too) but we >> can decide about it when we learn from the mixed-governance model and see if >> it works. >> >> * Problem 4: ACCEPTING NEW PROVIDERS >> >> If this is an acceptable approach - we can also apply a very similar >> governance model to adding new providers and that should unblock some of the >> PRs that are waiting for our decision. Knowing that we are going to split >> and that we can expect "commitment" from a stakeholder, we should be able to >> accept new providers. This might be possible assuming that the stakeholder >> will make a similar commitment - but for new providers, that commitment >> might also have to cover reviewing and testing new changes. We might also >> decide as a community to stop releasing new providers there if such support >> is missing. This way we can set the expectations we have as a community for >> new providers - we will release them as long as the stakeholder will >> actively make sure it is maintained. >> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >> >> We all know we want to split providers from core. By introducing >> mixed-governance we can test if it will work for the providers before we >> split them. It will take some time (and detailed AIP) to split, but in the >> meantime we can see if we will be able to apply the mixed-governance after >> the split. We will see if we can agree when it comes to expectations and >> find solutions before we actually split. >> >> J.
