I discussed this problem with Jarek. The group of stakeholders will be open and everyone can join, not just Google employees, etc. Release branches will be maintained in apache/airflow repository. Any non-committer change will still require PR. This means there is no vendor neutrality risk.
+1 I think this is a good step forward. pon., 20 cze 2022 o 21:18 Jarek Potiuk <[email protected]> napisał(a): > BTW. It will also be possible for anyone in the community to cherry-pick > changes from main and make a PR (which also a committer will have to > approve and merge). This is really no different that we have already done > with cherry-picked commits to "v1-10-stable" and "v-2-3-stable" branches by > non-committers. Random example here: > https://github.com/apache/airflow/pull/14090 > > We do not give any privileges to the organisations. Quite the opposite - > we make them responsible for preparing the PRs to be reviewed by committers. > > J. > > On Mon, Jun 20, 2022 at 9:07 PM Jarek Potiuk <[email protected]> wrote: > >> > I think we should continue to be strictly vendor-neutral. No >> organization should be able to gain special privileges or control a >> project’s direction. >> >> This is strictly vendor-neutral - Kamil - we are going to release the >> same changes that we are releasing already in main providers, just >> selectively cherry-picked (and then reviewed and merged by committer to the >> branch in airflow repo) - why do you think it is non-vendor neutral? >> >> On Mon, Jun 20, 2022 at 9:04 PM Jarek Potiuk <[email protected]> wrote: >> >>> > We can keep these branches in forks managed by stakeholders teams, but >>> I am afraid of the benefit that it will be then copied by us to our >>> repository and then released by us. If the release was prepared by an >>> external team, I think we should make it clear that it was prepared by >>> another team, including by publishing on the Pypi account of the team that >>> dealt with it. >>> >>> > Yes this is exactly what I proposed. >>> >>> Correction. I misread it. >>> >>> We are going to merge - only the cherry-picked changes that have been >>> reviewed and merged by the committer. Same way as today. Just the process >>> of cherry-picks is going to be done by the stakeholders, selecting the >>> things to cherry-pick (all the changes to cherry-pick should be already >>> merged in main). What we are going to do is to release subset of the >>> changes we already approved (and released - because we are going to release >>> those changes in the latest provider). So there will be no "new" changes in >>> those forks - those will be just cherry-picked changes reviewed by the >>> committer. There is no reason to mark it as "other" code - it will be the >>> same changes that are going to be released anyway (just a subset of those). >>> >>> J. >>> >>> On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <[email protected]> wrote: >>> >>>> > Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>> committers, because only they have write permission to the apache/airflow >>>> repository. As far as I know, Github does not allow us to grant write-only >>>> permissions to the selected branch. >>>> >>>> Kamil - you misunderstood it. The branch will be in the FORK of those >>>> users's choice - not in airflow repo. committer will merge that branch in >>>> the same way we do as today - but with fast-forwarding rather than >>>> squashing. >>>> >>>> > We can keep these branches in forks managed by stakeholders teams, >>>> but I am afraid of the benefit that it will be then copied by us to our >>>> repository and then released by us. If the release was prepared by an >>>> external team, I think we should make it clear that it was prepared by >>>> another team, including by publishing on the Pypi account of the team that >>>> dealt with it. >>>> >>>> Yes this is exactly what I proposed. >>>> >>>> > I think that everything Apache PMC releases should be prepared and >>>> created fully within the apache / airflow repositories. If stakeholders >>>> team do not have such a possibility, we should figure out that these teams >>>> become part of the community, and therefore work together with the entire >>>> community, not in isolation. Only then will we be able to act in accordance >>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>> particular each individual person will be able to contribute to the >>>> community as an individual, and not as a company or stakeholders team >>>> (Community of Peers) and no person will get special privileges just on the >>>> basis of their employment status (Earned Authority) >>>> >>>> Very true. And exactly follows my proposal :) >>>> >>>> J. >>>> >>>> >>>> On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <[email protected]> wrote: >>>> >>>>> +1 -- We have discussed this during the Airflow Summit in-person with >>>>> Ash, Rafal (and his team), Jarek and I about this for a long time, and I >>>>> think this is a good step forward. >>>>> >>>>> Regards, >>>>> Kaxil >>>>> >>>>> On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]> >>>>> wrote: >>>>> >>>>>> I think we should continue to be strictly vendor-neutral. No >>>>>> organization should be able to gain special privileges or control a >>>>>> project’s direction. >>>>>> >>>>>> pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]> >>>>>> napisał(a): >>>>>> >>>>>>> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>>>>> committers, because only they have write permission to the >>>>>>> apache/airflow >>>>>>> repository. As far as I know, Github does not allow us to grant >>>>>>> write-only >>>>>>> permissions to the selected branch. >>>>>>> >>>>>>> We can keep these branches in forks managed by stakeholders teams, >>>>>>> but I am afraid of the benefit that it will be then copied by us to our >>>>>>> repository and then released by us. If the release was prepared by an >>>>>>> external team, I think we should make it clear that it was prepared by >>>>>>> another team, including by publishing on the Pypi account of the team >>>>>>> that >>>>>>> dealt with it. >>>>>>> >>>>>>> I think that everything Apache PMC releases should be prepared and >>>>>>> created fully within the apache / airflow repositories. If stakeholders >>>>>>> team do not have such a possibility, we should figure out that these >>>>>>> teams >>>>>>> become part of the community, and therefore work together with the >>>>>>> entire >>>>>>> community, not in isolation. Only then will we be able to act in >>>>>>> accordance >>>>>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>>>>> particular each individual person will be able to contribute to the >>>>>>> community as an individual, and not as a company or stakeholders team >>>>>>> (Community of Peers) and no person will get special privileges just on >>>>>>> the >>>>>>> basis of their employment status (Earned Authority) >>>>>>> >>>>>>> >>>>>>> >>>>>>> pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]> >>>>>>> napisał(a): >>>>>>> >>>>>>>> > Will the people who maintain the providers' packages have the >>>>>>>> commiter >>>>>>>> >>>>>>>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>>>>>>> cherry-picking can be done in separate branches (and in this case in >>>>>>>> forks). >>>>>>>> Then the branch can be fast-forwarded by the committer in the >>>>>>>> "airflow" repo. No problem with that. >>>>>>>> >>>>>>>> J. >>>>>>>> >>>>>>>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Will the people who maintain the providers' packages have the >>>>>>>>> commiter >>>>>>>>> status? I guess it is necessary for people to have write access to >>>>>>>>> the >>>>>>>>> repository and therefore to be able to make cherry-pick changes to >>>>>>>>> the >>>>>>>>> branch. >>>>>>>>> >>>>>>>>> pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]> >>>>>>>>> napisał(a): >>>>>>>>> > >>>>>>>>> > +1 >>>>>>>>> > From my side the proposal handles all concerns I raised in >>>>>>>>> previous threads. >>>>>>>>> > I think mixed-governance is a step in the right direction. >>>>>>>>> > >>>>>>>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >> >>>>>>>>> >> Hello everyone, >>>>>>>>> >> >>>>>>>>> >> This is a follow-up after a few discussions started about >>>>>>>>> providers that were put on hold around the summit. I held a number of >>>>>>>>> discussions during theSummit and after, and as result I think I have a >>>>>>>>> proposal that can move forward some of the "stalled" decisions we >>>>>>>>> need to >>>>>>>>> make. >>>>>>>>> >> >>>>>>>>> >> TL;DR; >>>>>>>>> >> >>>>>>>>> >> My proposal is a "mixed-governance" model where "stakeholders" >>>>>>>>> are more responsible for cherry-picking and testing their providers >>>>>>>>> (including system testing) while Airflow PMC members will continue to >>>>>>>>> be >>>>>>>>> responsible for releasing them. >>>>>>>>> >> >>>>>>>>> >> Why do we need that? >>>>>>>>> >> >>>>>>>>> >> Google, Amazon and possibly others teams who are interested in >>>>>>>>> maintaining more backwards compatible versions of their providers will >>>>>>>>> commit to make PRs of the cherry-picks for older release branches of >>>>>>>>> their >>>>>>>>> providers. Those providers we release in parallel with the latest >>>>>>>>> versions >>>>>>>>> during the normal provider cycle. We can deprecate changes more >>>>>>>>> aggressively in the "latest" release if we do that. >>>>>>>>> >> >>>>>>>>> >> Those cherry-picked PRs will be driven, tested and performed by >>>>>>>>> the stakeholder teams (Google/Amazon, Databricks, others) and will >>>>>>>>> only >>>>>>>>> contain cherry-picks, while we - as PMC - will release them following >>>>>>>>> the >>>>>>>>> ASF rules (this is very important for the ASF to follow strict release >>>>>>>>> policies regarding who and how performs releases). >>>>>>>>> >> >>>>>>>>> >> This also allows us to introduce similar rules for new >>>>>>>>> provider's acceptance for new providers for "main releases". It also >>>>>>>>> allows >>>>>>>>> running the "system tests" for the provider under control of the >>>>>>>>> stakeholder (after applying AIP-47 changes). >>>>>>>>> >> >>>>>>>>> >> Example 1: Google team can cherry-pick changes to a >>>>>>>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>>>>>>> provider with some of the bug-fixes and features (together with - say >>>>>>>>> latest 8.1.0). >>>>>>>>> >> >>>>>>>>> >> Example 2: DataLake provider from Databricks - can get accepted >>>>>>>>> if Databricks commits to maintaining it. We will release the provider >>>>>>>>> as >>>>>>>>> long as Databricks maintains it. >>>>>>>>> >> >>>>>>>>> >> Longer context: >>>>>>>>> >> >>>>>>>>> >> I have - in my mind so far - a longer roadmap for providers >>>>>>>>> that will lead them to be separated from the core and I want to write >>>>>>>>> an >>>>>>>>> AIP about that soon. This AIP will detail all the steps needed - I >>>>>>>>> will >>>>>>>>> work with multiple interested parties on it and it will take some >>>>>>>>> time to >>>>>>>>> agree and complete. But I want to start with something tangible that >>>>>>>>> will >>>>>>>>> solve quite a few problems that were raised recently and something >>>>>>>>> that >>>>>>>>> seems to be possible to be solved in the current provider release >>>>>>>>> cycle >>>>>>>>> (till the end of June) and test some of the governance approach. >>>>>>>>> >> >>>>>>>>> >> This proposal simply builds on our semver approach - we do not >>>>>>>>> change it, we just start releasing some providers (those that have >>>>>>>>> some >>>>>>>>> backing from stakeholders) in more than one version - including >>>>>>>>> "latest" >>>>>>>>> and earlier, more backwards-compatible branches. Not all providers - >>>>>>>>> just >>>>>>>>> some. Not all branches - just those that the stakeholders will commit >>>>>>>>> to >>>>>>>>> maintain. >>>>>>>>> >> >>>>>>>>> >> We need such a commitment from stakeholders, because we - as >>>>>>>>> the Airflow community and maintainers, want to only actively >>>>>>>>> maintain the >>>>>>>>> latest releases, where it is in the interest of the stakeholders to >>>>>>>>> cherry-pick and test also earlier, more backwards compatible releases >>>>>>>>> of >>>>>>>>> their choice. >>>>>>>>> >> >>>>>>>>> >> What problems this proposal solves: >>>>>>>>> >> >>>>>>>>> >> * Problem 1: DEPRECATION REMOVAL >>>>>>>>> >> >>>>>>>>> >> we can remove deprecations faster in "main" versions of the >>>>>>>>> providers - no need to introduce a deprecation policy - the >>>>>>>>> stakeholders >>>>>>>>> for the providers will take care about cherry-picking and maintaining >>>>>>>>> more >>>>>>>>> "backwards-compatible" versions. We are free to remove deprecations in >>>>>>>>> major releases (in a cherry-pickable way of course). >>>>>>>>> >> >>>>>>>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>>>>>>> >> >>>>>>>>> >> We avoid the problem (already happened with the composer >>>>>>>>> release) that the stakeholders in a given provider had to release >>>>>>>>> their own >>>>>>>>> version which was not available in their community - with some >>>>>>>>> cherry-picks. We want to avoid "diverging" there - by releasing the >>>>>>>>> cherry-picked providers by the community, we also give other users an >>>>>>>>> opportunity to follow "slower" deprecation policies for as long as it >>>>>>>>> is >>>>>>>>> maintained. >>>>>>>>> >> >>>>>>>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>>>>>>> >> >>>>>>>>> >> We are going to test a governance model that we might apply >>>>>>>>> when we split providers. We are talking about it for quite some time >>>>>>>>> - but >>>>>>>>> this is what helps us to test the model where stakeholders provide >>>>>>>>> more >>>>>>>>> "maintenance" while the community still takes care about releases. We >>>>>>>>> (as >>>>>>>>> community) can commit to releasing such a version of a provider as >>>>>>>>> long as >>>>>>>>> the stakeholder will actively maintain it. We can stop at any moment >>>>>>>>> if we >>>>>>>>> do not have support from the stakeholder. If it works - we can keep >>>>>>>>> it as a >>>>>>>>> long-term solution. In the future we can think of other scenarios >>>>>>>>> (passing >>>>>>>>> ownership of a provider to stakeholders who want it - providing we >>>>>>>>> want it >>>>>>>>> too) but we can decide about it when we learn from the >>>>>>>>> mixed-governance >>>>>>>>> model and see if it works. >>>>>>>>> >> >>>>>>>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>>>>>>> >> >>>>>>>>> >> If this is an acceptable approach - we can also apply a very >>>>>>>>> similar governance model to adding new providers and that should >>>>>>>>> unblock >>>>>>>>> some of the PRs that are waiting for our decision. Knowing that we are >>>>>>>>> going to split and that we can expect "commitment" from a >>>>>>>>> stakeholder, we >>>>>>>>> should be able to accept new providers. This might be possible >>>>>>>>> assuming >>>>>>>>> that the stakeholder will make a similar commitment - but for new >>>>>>>>> providers, that commitment might also have to cover reviewing and >>>>>>>>> testing >>>>>>>>> new changes. We might also decide as a community to stop releasing new >>>>>>>>> providers there if such support is missing. This way we can set the >>>>>>>>> expectations we have as a community for new providers - we will >>>>>>>>> release >>>>>>>>> them as long as the stakeholder will actively make sure it is >>>>>>>>> maintained. >>>>>>>>> >> >>>>>>>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>>>>>>> >> >>>>>>>>> >> We all know we want to split providers from core. By >>>>>>>>> introducing mixed-governance we can test if it will work for the >>>>>>>>> providers >>>>>>>>> before we split them. It will take some time (and detailed AIP) to >>>>>>>>> split, >>>>>>>>> but in the meantime we can see if we will be able to apply the >>>>>>>>> mixed-governance after the split. We will see if we can agree when it >>>>>>>>> comes >>>>>>>>> to expectations and find solutions before we actually split. >>>>>>>>> >> >>>>>>>>> >> J. >>>>>>>>> >>>>>>>>
