> I think we should continue to be strictly vendor-neutral. No organization should be able to gain special privileges or control a project’s direction.
This is strictly vendor-neutral - Kamil - we are going to release the same changes that we are releasing already in main providers, just selectively cherry-picked (and then reviewed and merged by committer to the branch in airflow repo) - why do you think it is non-vendor neutral? On Mon, Jun 20, 2022 at 9:04 PM Jarek Potiuk <[email protected]> wrote: > > We can keep these branches in forks managed by stakeholders teams, but I > am afraid of the benefit that it will be then copied by us to our > repository and then released by us. If the release was prepared by an > external team, I think we should make it clear that it was prepared by > another team, including by publishing on the Pypi account of the team that > dealt with it. > > > Yes this is exactly what I proposed. > > Correction. I misread it. > > We are going to merge - only the cherry-picked changes that have been > reviewed and merged by the committer. Same way as today. Just the process > of cherry-picks is going to be done by the stakeholders, selecting the > things to cherry-pick (all the changes to cherry-pick should be already > merged in main). What we are going to do is to release subset of the > changes we already approved (and released - because we are going to release > those changes in the latest provider). So there will be no "new" changes in > those forks - those will be just cherry-picked changes reviewed by the > committer. There is no reason to mark it as "other" code - it will be the > same changes that are going to be released anyway (just a subset of those). > > J. > > On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <[email protected]> wrote: > >> > Cherry-picking to branch v2-2* or 1.10.* can only be done by the >> committers, because only they have write permission to the apache/airflow >> repository. As far as I know, Github does not allow us to grant write-only >> permissions to the selected branch. >> >> Kamil - you misunderstood it. The branch will be in the FORK of those >> users's choice - not in airflow repo. committer will merge that branch in >> the same way we do as today - but with fast-forwarding rather than >> squashing. >> >> > We can keep these branches in forks managed by stakeholders teams, but >> I am afraid of the benefit that it will be then copied by us to our >> repository and then released by us. If the release was prepared by an >> external team, I think we should make it clear that it was prepared by >> another team, including by publishing on the Pypi account of the team that >> dealt with it. >> >> Yes this is exactly what I proposed. >> >> > I think that everything Apache PMC releases should be prepared and >> created fully within the apache / airflow repositories. If stakeholders >> team do not have such a possibility, we should figure out that these teams >> become part of the community, and therefore work together with the entire >> community, not in isolation. Only then will we be able to act in accordance >> with the Apache Way <https://www.apache.org/theapacheway/>, in >> particular each individual person will be able to contribute to the >> community as an individual, and not as a company or stakeholders team >> (Community of Peers) and no person will get special privileges just on the >> basis of their employment status (Earned Authority) >> >> Very true. And exactly follows my proposal :) >> >> J. >> >> >> On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <[email protected]> wrote: >> >>> +1 -- We have discussed this during the Airflow Summit in-person with >>> Ash, Rafal (and his team), Jarek and I about this for a long time, and I >>> think this is a good step forward. >>> >>> Regards, >>> Kaxil >>> >>> On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]> wrote: >>> >>>> I think we should continue to be strictly vendor-neutral. No >>>> organization should be able to gain special privileges or control a >>>> project’s direction. >>>> >>>> pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]> >>>> napisał(a): >>>> >>>>> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>>> committers, because only they have write permission to the apache/airflow >>>>> repository. As far as I know, Github does not allow us to grant write-only >>>>> permissions to the selected branch. >>>>> >>>>> We can keep these branches in forks managed by stakeholders teams, but >>>>> I am afraid of the benefit that it will be then copied by us to our >>>>> repository and then released by us. If the release was prepared by an >>>>> external team, I think we should make it clear that it was prepared by >>>>> another team, including by publishing on the Pypi account of the team that >>>>> dealt with it. >>>>> >>>>> I think that everything Apache PMC releases should be prepared and >>>>> created fully within the apache / airflow repositories. If stakeholders >>>>> team do not have such a possibility, we should figure out that these teams >>>>> become part of the community, and therefore work together with the entire >>>>> community, not in isolation. Only then will we be able to act in >>>>> accordance >>>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>>> particular each individual person will be able to contribute to the >>>>> community as an individual, and not as a company or stakeholders team >>>>> (Community of Peers) and no person will get special privileges just on the >>>>> basis of their employment status (Earned Authority) >>>>> >>>>> >>>>> >>>>> pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]> napisał(a): >>>>> >>>>>> > Will the people who maintain the providers' packages have the >>>>>> commiter >>>>>> >>>>>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>>>>> cherry-picking can be done in separate branches (and in this case in >>>>>> forks). >>>>>> Then the branch can be fast-forwarded by the committer in the >>>>>> "airflow" repo. No problem with that. >>>>>> >>>>>> J. >>>>>> >>>>>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Will the people who maintain the providers' packages have the >>>>>>> commiter >>>>>>> status? I guess it is necessary for people to have write access to >>>>>>> the >>>>>>> repository and therefore to be able to make cherry-pick changes to >>>>>>> the >>>>>>> branch. >>>>>>> >>>>>>> pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]> >>>>>>> napisał(a): >>>>>>> > >>>>>>> > +1 >>>>>>> > From my side the proposal handles all concerns I raised in >>>>>>> previous threads. >>>>>>> > I think mixed-governance is a step in the right direction. >>>>>>> > >>>>>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> Hello everyone, >>>>>>> >> >>>>>>> >> This is a follow-up after a few discussions started about >>>>>>> providers that were put on hold around the summit. I held a number of >>>>>>> discussions during theSummit and after, and as result I think I have a >>>>>>> proposal that can move forward some of the "stalled" decisions we need >>>>>>> to >>>>>>> make. >>>>>>> >> >>>>>>> >> TL;DR; >>>>>>> >> >>>>>>> >> My proposal is a "mixed-governance" model where "stakeholders" >>>>>>> are more responsible for cherry-picking and testing their providers >>>>>>> (including system testing) while Airflow PMC members will continue to be >>>>>>> responsible for releasing them. >>>>>>> >> >>>>>>> >> Why do we need that? >>>>>>> >> >>>>>>> >> Google, Amazon and possibly others teams who are interested in >>>>>>> maintaining more backwards compatible versions of their providers will >>>>>>> commit to make PRs of the cherry-picks for older release branches of >>>>>>> their >>>>>>> providers. Those providers we release in parallel with the latest >>>>>>> versions >>>>>>> during the normal provider cycle. We can deprecate changes more >>>>>>> aggressively in the "latest" release if we do that. >>>>>>> >> >>>>>>> >> Those cherry-picked PRs will be driven, tested and performed by >>>>>>> the stakeholder teams (Google/Amazon, Databricks, others) and will only >>>>>>> contain cherry-picks, while we - as PMC - will release them following >>>>>>> the >>>>>>> ASF rules (this is very important for the ASF to follow strict release >>>>>>> policies regarding who and how performs releases). >>>>>>> >> >>>>>>> >> This also allows us to introduce similar rules for new provider's >>>>>>> acceptance for new providers for "main releases". It also allows running >>>>>>> the "system tests" for the provider under control of the stakeholder >>>>>>> (after >>>>>>> applying AIP-47 changes). >>>>>>> >> >>>>>>> >> Example 1: Google team can cherry-pick changes to a >>>>>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>>>>> provider with some of the bug-fixes and features (together with - say >>>>>>> latest 8.1.0). >>>>>>> >> >>>>>>> >> Example 2: DataLake provider from Databricks - can get accepted >>>>>>> if Databricks commits to maintaining it. We will release the provider as >>>>>>> long as Databricks maintains it. >>>>>>> >> >>>>>>> >> Longer context: >>>>>>> >> >>>>>>> >> I have - in my mind so far - a longer roadmap for providers that >>>>>>> will lead them to be separated from the core and I want to write an AIP >>>>>>> about that soon. This AIP will detail all the steps needed - I will work >>>>>>> with multiple interested parties on it and it will take some time to >>>>>>> agree >>>>>>> and complete. But I want to start with something tangible that will >>>>>>> solve >>>>>>> quite a few problems that were raised recently and something that seems >>>>>>> to >>>>>>> be possible to be solved in the current provider release cycle (till the >>>>>>> end of June) and test some of the governance approach. >>>>>>> >> >>>>>>> >> This proposal simply builds on our semver approach - we do not >>>>>>> change it, we just start releasing some providers (those that have some >>>>>>> backing from stakeholders) in more than one version - including "latest" >>>>>>> and earlier, more backwards-compatible branches. Not all providers - >>>>>>> just >>>>>>> some. Not all branches - just those that the stakeholders will commit to >>>>>>> maintain. >>>>>>> >> >>>>>>> >> We need such a commitment from stakeholders, because we - as the >>>>>>> Airflow community and maintainers, want to only actively maintain the >>>>>>> latest releases, where it is in the interest of the stakeholders to >>>>>>> cherry-pick and test also earlier, more backwards compatible releases of >>>>>>> their choice. >>>>>>> >> >>>>>>> >> What problems this proposal solves: >>>>>>> >> >>>>>>> >> * Problem 1: DEPRECATION REMOVAL >>>>>>> >> >>>>>>> >> we can remove deprecations faster in "main" versions of the >>>>>>> providers - no need to introduce a deprecation policy - the stakeholders >>>>>>> for the providers will take care about cherry-picking and maintaining >>>>>>> more >>>>>>> "backwards-compatible" versions. We are free to remove deprecations in >>>>>>> major releases (in a cherry-pickable way of course). >>>>>>> >> >>>>>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>>>>> >> >>>>>>> >> We avoid the problem (already happened with the composer release) >>>>>>> that the stakeholders in a given provider had to release their own >>>>>>> version >>>>>>> which was not available in their community - with some cherry-picks. We >>>>>>> want to avoid "diverging" there - by releasing the cherry-picked >>>>>>> providers >>>>>>> by the community, we also give other users an opportunity to follow >>>>>>> "slower" deprecation policies for as long as it is maintained. >>>>>>> >> >>>>>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>>>>> >> >>>>>>> >> We are going to test a governance model that we might apply when >>>>>>> we split providers. We are talking about it for quite some time - but >>>>>>> this >>>>>>> is what helps us to test the model where stakeholders provide more >>>>>>> "maintenance" while the community still takes care about releases. We >>>>>>> (as >>>>>>> community) can commit to releasing such a version of a provider as long >>>>>>> as >>>>>>> the stakeholder will actively maintain it. We can stop at any moment if >>>>>>> we >>>>>>> do not have support from the stakeholder. If it works - we can keep it >>>>>>> as a >>>>>>> long-term solution. In the future we can think of other scenarios >>>>>>> (passing >>>>>>> ownership of a provider to stakeholders who want it - providing we want >>>>>>> it >>>>>>> too) but we can decide about it when we learn from the mixed-governance >>>>>>> model and see if it works. >>>>>>> >> >>>>>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>>>>> >> >>>>>>> >> If this is an acceptable approach - we can also apply a very >>>>>>> similar governance model to adding new providers and that should unblock >>>>>>> some of the PRs that are waiting for our decision. Knowing that we are >>>>>>> going to split and that we can expect "commitment" from a stakeholder, >>>>>>> we >>>>>>> should be able to accept new providers. This might be possible assuming >>>>>>> that the stakeholder will make a similar commitment - but for new >>>>>>> providers, that commitment might also have to cover reviewing and >>>>>>> testing >>>>>>> new changes. We might also decide as a community to stop releasing new >>>>>>> providers there if such support is missing. This way we can set the >>>>>>> expectations we have as a community for new providers - we will release >>>>>>> them as long as the stakeholder will actively make sure it is >>>>>>> maintained. >>>>>>> >> >>>>>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>>>>> >> >>>>>>> >> We all know we want to split providers from core. By introducing >>>>>>> mixed-governance we can test if it will work for the providers before we >>>>>>> split them. It will take some time (and detailed AIP) to split, but in >>>>>>> the >>>>>>> meantime we can see if we will be able to apply the mixed-governance >>>>>>> after >>>>>>> the split. We will see if we can agree when it comes to expectations and >>>>>>> find solutions before we actually split. >>>>>>> >> >>>>>>> >> J. >>>>>>> >>>>>>
