> +1 I agree this is a logical next step towards possibly separating provider code from the Airflow code base (and it's useful even if we never do that).
I am quite sure we will :) On Thu, Jun 23, 2022 at 1:18 AM Oliveira, Niko <[email protected]> wrote: > +1 I agree this is a logical next step towards possibly separating > provider code from the Airflow code base (and it's useful even if we never > do that). > > Cheers, > Niko > ------------------------------ > *From:* Kamil Breguła <[email protected]> > *Sent:* Monday, June 20, 2022 1:30 PM > *To:* [email protected] > *Subject:* RE: [EXTERNAL][PROPOSAL] Provider's mixed governance model - > first step of provider separation > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > I discussed this problem with Jarek. The group of stakeholders will be > open and everyone can join, not just Google employees, etc. Release > branches will be maintained in apache/airflow repository. Any non-committer > change will still require PR. This means there is no vendor neutrality risk. > > +1 I think this is a good step forward. > > pon., 20 cze 2022 o 21:18 Jarek Potiuk <[email protected]> napisał(a): > >> BTW. It will also be possible for anyone in the community to cherry-pick >> changes from main and make a PR (which also a committer will have to >> approve and merge). This is really no different that we have already done >> with cherry-picked commits to "v1-10-stable" and "v-2-3-stable" branches by >> non-committers. Random example here: >> https://github.com/apache/airflow/pull/14090 >> >> We do not give any privileges to the organisations. Quite the opposite - >> we make them responsible for preparing the PRs to be reviewed by committers. >> >> J. >> >> On Mon, Jun 20, 2022 at 9:07 PM Jarek Potiuk <[email protected]> wrote: >> >>> > I think we should continue to be strictly vendor-neutral. No >>> organization should be able to gain special privileges or control a >>> project’s direction. >>> >>> This is strictly vendor-neutral - Kamil - we are going to release the >>> same changes that we are releasing already in main providers, just >>> selectively cherry-picked (and then reviewed and merged by committer to the >>> branch in airflow repo) - why do you think it is non-vendor neutral? >>> >>> On Mon, Jun 20, 2022 at 9:04 PM Jarek Potiuk <[email protected]> wrote: >>> >>>> > We can keep these branches in forks managed by stakeholders teams, >>>> but I am afraid of the benefit that it will be then copied by us to our >>>> repository and then released by us. If the release was prepared by an >>>> external team, I think we should make it clear that it was prepared by >>>> another team, including by publishing on the Pypi account of the team that >>>> dealt with it. >>>> >>>> > Yes this is exactly what I proposed. >>>> >>>> Correction. I misread it. >>>> >>>> We are going to merge - only the cherry-picked changes that have been >>>> reviewed and merged by the committer. Same way as today. Just the process >>>> of cherry-picks is going to be done by the stakeholders, selecting the >>>> things to cherry-pick (all the changes to cherry-pick should be already >>>> merged in main). What we are going to do is to release subset of the >>>> changes we already approved (and released - because we are going to release >>>> those changes in the latest provider). So there will be no "new" changes in >>>> those forks - those will be just cherry-picked changes reviewed by the >>>> committer. There is no reason to mark it as "other" code - it will be the >>>> same changes that are going to be released anyway (just a subset of those). >>>> >>>> J. >>>> >>>> On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <[email protected]> wrote: >>>> >>>>> > Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>>> committers, because only they have write permission to the apache/airflow >>>>> repository. As far as I know, Github does not allow us to grant write-only >>>>> permissions to the selected branch. >>>>> >>>>> Kamil - you misunderstood it. The branch will be in the FORK of those >>>>> users's choice - not in airflow repo. committer will merge that branch in >>>>> the same way we do as today - but with fast-forwarding rather than >>>>> squashing. >>>>> >>>>> > We can keep these branches in forks managed by stakeholders teams, >>>>> but I am afraid of the benefit that it will be then copied by us to our >>>>> repository and then released by us. If the release was prepared by an >>>>> external team, I think we should make it clear that it was prepared by >>>>> another team, including by publishing on the Pypi account of the team that >>>>> dealt with it. >>>>> >>>>> Yes this is exactly what I proposed. >>>>> >>>>> > I think that everything Apache PMC releases should be prepared and >>>>> created fully within the apache / airflow repositories. If stakeholders >>>>> team do not have such a possibility, we should figure out that these teams >>>>> become part of the community, and therefore work together with the entire >>>>> community, not in isolation. Only then will we be able to act in >>>>> accordance >>>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>>> particular each individual person will be able to contribute to the >>>>> community as an individual, and not as a company or stakeholders team >>>>> (Community of Peers) and no person will get special privileges just on the >>>>> basis of their employment status (Earned Authority) >>>>> >>>>> Very true. And exactly follows my proposal :) >>>>> >>>>> J. >>>>> >>>>> >>>>> On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <[email protected]> >>>>> wrote: >>>>> >>>>>> +1 -- We have discussed this during the Airflow Summit in-person with >>>>>> Ash, Rafal (and his team), Jarek and I about this for a long time, and I >>>>>> think this is a good step forward. >>>>>> >>>>>> Regards, >>>>>> Kaxil >>>>>> >>>>>> On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I think we should continue to be strictly vendor-neutral. No >>>>>>> organization should be able to gain special privileges or control a >>>>>>> project’s direction. >>>>>>> >>>>>>> pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]> >>>>>>> napisał(a): >>>>>>> >>>>>>>> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>>>>>> committers, because only they have write permission to the >>>>>>>> apache/airflow >>>>>>>> repository. As far as I know, Github does not allow us to grant >>>>>>>> write-only >>>>>>>> permissions to the selected branch. >>>>>>>> >>>>>>>> We can keep these branches in forks managed by stakeholders teams, >>>>>>>> but I am afraid of the benefit that it will be then copied by us to our >>>>>>>> repository and then released by us. If the release was prepared by an >>>>>>>> external team, I think we should make it clear that it was prepared by >>>>>>>> another team, including by publishing on the Pypi account of the team >>>>>>>> that >>>>>>>> dealt with it. >>>>>>>> >>>>>>>> I think that everything Apache PMC releases should be prepared and >>>>>>>> created fully within the apache / airflow repositories. If stakeholders >>>>>>>> team do not have such a possibility, we should figure out that these >>>>>>>> teams >>>>>>>> become part of the community, and therefore work together with the >>>>>>>> entire >>>>>>>> community, not in isolation. Only then will we be able to act in >>>>>>>> accordance >>>>>>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>>>>>> particular each individual person will be able to contribute to the >>>>>>>> community as an individual, and not as a company or stakeholders team >>>>>>>> (Community of Peers) and no person will get special privileges just on >>>>>>>> the >>>>>>>> basis of their employment status (Earned Authority) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]> >>>>>>>> napisał(a): >>>>>>>> >>>>>>>>> > Will the people who maintain the providers' packages have the >>>>>>>>> commiter >>>>>>>>> >>>>>>>>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>>>>>>>> cherry-picking can be done in separate branches (and in this case in >>>>>>>>> forks). >>>>>>>>> Then the branch can be fast-forwarded by the committer in the >>>>>>>>> "airflow" repo. No problem with that. >>>>>>>>> >>>>>>>>> J. >>>>>>>>> >>>>>>>>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Will the people who maintain the providers' packages have the >>>>>>>>>> commiter >>>>>>>>>> status? I guess it is necessary for people to have write access >>>>>>>>>> to the >>>>>>>>>> repository and therefore to be able to make cherry-pick changes >>>>>>>>>> to the >>>>>>>>>> branch. >>>>>>>>>> >>>>>>>>>> pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]> >>>>>>>>>> napisał(a): >>>>>>>>>> > >>>>>>>>>> > +1 >>>>>>>>>> > From my side the proposal handles all concerns I raised in >>>>>>>>>> previous threads. >>>>>>>>>> > I think mixed-governance is a step in the right direction. >>>>>>>>>> > >>>>>>>>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Hello everyone, >>>>>>>>>> >> >>>>>>>>>> >> This is a follow-up after a few discussions started about >>>>>>>>>> providers that were put on hold around the summit. I held a number >>>>>>>>>> of >>>>>>>>>> discussions during theSummit and after, and as result I think I have >>>>>>>>>> a >>>>>>>>>> proposal that can move forward some of the "stalled" decisions we >>>>>>>>>> need to >>>>>>>>>> make. >>>>>>>>>> >> >>>>>>>>>> >> TL;DR; >>>>>>>>>> >> >>>>>>>>>> >> My proposal is a "mixed-governance" model where "stakeholders" >>>>>>>>>> are more responsible for cherry-picking and testing their providers >>>>>>>>>> (including system testing) while Airflow PMC members will continue >>>>>>>>>> to be >>>>>>>>>> responsible for releasing them. >>>>>>>>>> >> >>>>>>>>>> >> Why do we need that? >>>>>>>>>> >> >>>>>>>>>> >> Google, Amazon and possibly others teams who are interested in >>>>>>>>>> maintaining more backwards compatible versions of their providers >>>>>>>>>> will >>>>>>>>>> commit to make PRs of the cherry-picks for older release branches of >>>>>>>>>> their >>>>>>>>>> providers. Those providers we release in parallel with the latest >>>>>>>>>> versions >>>>>>>>>> during the normal provider cycle. We can deprecate changes more >>>>>>>>>> aggressively in the "latest" release if we do that. >>>>>>>>>> >> >>>>>>>>>> >> Those cherry-picked PRs will be driven, tested and performed >>>>>>>>>> by the stakeholder teams (Google/Amazon, Databricks, others) and >>>>>>>>>> will only >>>>>>>>>> contain cherry-picks, while we - as PMC - will release them >>>>>>>>>> following the >>>>>>>>>> ASF rules (this is very important for the ASF to follow strict >>>>>>>>>> release >>>>>>>>>> policies regarding who and how performs releases). >>>>>>>>>> >> >>>>>>>>>> >> This also allows us to introduce similar rules for new >>>>>>>>>> provider's acceptance for new providers for "main releases". It also >>>>>>>>>> allows >>>>>>>>>> running the "system tests" for the provider under control of the >>>>>>>>>> stakeholder (after applying AIP-47 changes). >>>>>>>>>> >> >>>>>>>>>> >> Example 1: Google team can cherry-pick changes to a >>>>>>>>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>>>>>>>> provider with some of the bug-fixes and features (together with - say >>>>>>>>>> latest 8.1.0). >>>>>>>>>> >> >>>>>>>>>> >> Example 2: DataLake provider from Databricks - can get >>>>>>>>>> accepted if Databricks commits to maintaining it. We will release the >>>>>>>>>> provider as long as Databricks maintains it. >>>>>>>>>> >> >>>>>>>>>> >> Longer context: >>>>>>>>>> >> >>>>>>>>>> >> I have - in my mind so far - a longer roadmap for providers >>>>>>>>>> that will lead them to be separated from the core and I want to >>>>>>>>>> write an >>>>>>>>>> AIP about that soon. This AIP will detail all the steps needed - I >>>>>>>>>> will >>>>>>>>>> work with multiple interested parties on it and it will take some >>>>>>>>>> time to >>>>>>>>>> agree and complete. But I want to start with something tangible >>>>>>>>>> that will >>>>>>>>>> solve quite a few problems that were raised recently and something >>>>>>>>>> that >>>>>>>>>> seems to be possible to be solved in the current provider release >>>>>>>>>> cycle >>>>>>>>>> (till the end of June) and test some of the governance approach. >>>>>>>>>> >> >>>>>>>>>> >> This proposal simply builds on our semver approach - we do not >>>>>>>>>> change it, we just start releasing some providers (those that have >>>>>>>>>> some >>>>>>>>>> backing from stakeholders) in more than one version - including >>>>>>>>>> "latest" >>>>>>>>>> and earlier, more backwards-compatible branches. Not all providers - >>>>>>>>>> just >>>>>>>>>> some. Not all branches - just those that the stakeholders will >>>>>>>>>> commit to >>>>>>>>>> maintain. >>>>>>>>>> >> >>>>>>>>>> >> We need such a commitment from stakeholders, because we - as >>>>>>>>>> the Airflow community and maintainers, want to only actively >>>>>>>>>> maintain the >>>>>>>>>> latest releases, where it is in the interest of the stakeholders to >>>>>>>>>> cherry-pick and test also earlier, more backwards compatible >>>>>>>>>> releases of >>>>>>>>>> their choice. >>>>>>>>>> >> >>>>>>>>>> >> What problems this proposal solves: >>>>>>>>>> >> >>>>>>>>>> >> * Problem 1: DEPRECATION REMOVAL >>>>>>>>>> >> >>>>>>>>>> >> we can remove deprecations faster in "main" versions of the >>>>>>>>>> providers - no need to introduce a deprecation policy - the >>>>>>>>>> stakeholders >>>>>>>>>> for the providers will take care about cherry-picking and >>>>>>>>>> maintaining more >>>>>>>>>> "backwards-compatible" versions. We are free to remove deprecations >>>>>>>>>> in >>>>>>>>>> major releases (in a cherry-pickable way of course). >>>>>>>>>> >> >>>>>>>>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>>>>>>>> >> >>>>>>>>>> >> We avoid the problem (already happened with the composer >>>>>>>>>> release) that the stakeholders in a given provider had to release >>>>>>>>>> their own >>>>>>>>>> version which was not available in their community - with some >>>>>>>>>> cherry-picks. We want to avoid "diverging" there - by releasing the >>>>>>>>>> cherry-picked providers by the community, we also give other users an >>>>>>>>>> opportunity to follow "slower" deprecation policies for as long as >>>>>>>>>> it is >>>>>>>>>> maintained. >>>>>>>>>> >> >>>>>>>>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>>>>>>>> >> >>>>>>>>>> >> We are going to test a governance model that we might apply >>>>>>>>>> when we split providers. We are talking about it for quite some time >>>>>>>>>> - but >>>>>>>>>> this is what helps us to test the model where stakeholders provide >>>>>>>>>> more >>>>>>>>>> "maintenance" while the community still takes care about releases. >>>>>>>>>> We (as >>>>>>>>>> community) can commit to releasing such a version of a provider as >>>>>>>>>> long as >>>>>>>>>> the stakeholder will actively maintain it. We can stop at any moment >>>>>>>>>> if we >>>>>>>>>> do not have support from the stakeholder. If it works - we can keep >>>>>>>>>> it as a >>>>>>>>>> long-term solution. In the future we can think of other scenarios >>>>>>>>>> (passing >>>>>>>>>> ownership of a provider to stakeholders who want it - providing we >>>>>>>>>> want it >>>>>>>>>> too) but we can decide about it when we learn from the >>>>>>>>>> mixed-governance >>>>>>>>>> model and see if it works. >>>>>>>>>> >> >>>>>>>>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>>>>>>>> >> >>>>>>>>>> >> If this is an acceptable approach - we can also apply a very >>>>>>>>>> similar governance model to adding new providers and that should >>>>>>>>>> unblock >>>>>>>>>> some of the PRs that are waiting for our decision. Knowing that we >>>>>>>>>> are >>>>>>>>>> going to split and that we can expect "commitment" from a >>>>>>>>>> stakeholder, we >>>>>>>>>> should be able to accept new providers. This might be possible >>>>>>>>>> assuming >>>>>>>>>> that the stakeholder will make a similar commitment - but for new >>>>>>>>>> providers, that commitment might also have to cover reviewing and >>>>>>>>>> testing >>>>>>>>>> new changes. We might also decide as a community to stop releasing >>>>>>>>>> new >>>>>>>>>> providers there if such support is missing. This way we can set the >>>>>>>>>> expectations we have as a community for new providers - we will >>>>>>>>>> release >>>>>>>>>> them as long as the stakeholder will actively make sure it is >>>>>>>>>> maintained. >>>>>>>>>> >> >>>>>>>>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>>>>>>>> >> >>>>>>>>>> >> We all know we want to split providers from core. By >>>>>>>>>> introducing mixed-governance we can test if it will work for the >>>>>>>>>> providers >>>>>>>>>> before we split them. It will take some time (and detailed AIP) to >>>>>>>>>> split, >>>>>>>>>> but in the meantime we can see if we will be able to apply the >>>>>>>>>> mixed-governance after the split. We will see if we can agree when >>>>>>>>>> it comes >>>>>>>>>> to expectations and find solutions before we actually split. >>>>>>>>>> >> >>>>>>>>>> >> J. >>>>>>>>>> >>>>>>>>>
