> We can keep these branches in forks managed by stakeholders teams, but I am afraid of the benefit that it will be then copied by us to our repository and then released by us. If the release was prepared by an external team, I think we should make it clear that it was prepared by another team, including by publishing on the Pypi account of the team that dealt with it.
> Yes this is exactly what I proposed. Correction. I misread it. We are going to merge - only the cherry-picked changes that have been reviewed and merged by the committer. Same way as today. Just the process of cherry-picks is going to be done by the stakeholders, selecting the things to cherry-pick (all the changes to cherry-pick should be already merged in main). What we are going to do is to release subset of the changes we already approved (and released - because we are going to release those changes in the latest provider). So there will be no "new" changes in those forks - those will be just cherry-picked changes reviewed by the committer. There is no reason to mark it as "other" code - it will be the same changes that are going to be released anyway (just a subset of those). J. On Mon, Jun 20, 2022 at 9:00 PM Jarek Potiuk <[email protected]> wrote: > > Cherry-picking to branch v2-2* or 1.10.* can only be done by the > committers, because only they have write permission to the apache/airflow > repository. As far as I know, Github does not allow us to grant write-only > permissions to the selected branch. > > Kamil - you misunderstood it. The branch will be in the FORK of those > users's choice - not in airflow repo. committer will merge that branch in > the same way we do as today - but with fast-forwarding rather than > squashing. > > > We can keep these branches in forks managed by stakeholders teams, but I > am afraid of the benefit that it will be then copied by us to our > repository and then released by us. If the release was prepared by an > external team, I think we should make it clear that it was prepared by > another team, including by publishing on the Pypi account of the team that > dealt with it. > > Yes this is exactly what I proposed. > > > I think that everything Apache PMC releases should be prepared and > created fully within the apache / airflow repositories. If stakeholders > team do not have such a possibility, we should figure out that these teams > become part of the community, and therefore work together with the entire > community, not in isolation. Only then will we be able to act in accordance > with the Apache Way <https://www.apache.org/theapacheway/>, in particular > each individual person will be able to contribute to the community as an > individual, and not as a company or stakeholders team (Community of Peers) > and no person will get special privileges just on the basis of their > employment status (Earned Authority) > > Very true. And exactly follows my proposal :) > > J. > > > On Mon, Jun 20, 2022 at 8:16 PM Kaxil Naik <[email protected]> wrote: > >> +1 -- We have discussed this during the Airflow Summit in-person with >> Ash, Rafal (and his team), Jarek and I about this for a long time, and I >> think this is a good step forward. >> >> Regards, >> Kaxil >> >> On Mon, 20 Jun 2022 at 17:26, Kamil Breguła <[email protected]> wrote: >> >>> I think we should continue to be strictly vendor-neutral. No >>> organization should be able to gain special privileges or control a >>> project’s direction. >>> >>> pon., 20 cze 2022 o 18:14 Kamil Breguła <[email protected]> napisał(a): >>> >>>> Cherry-picking to branch v2-2* or 1.10.* can only be done by the >>>> committers, because only they have write permission to the apache/airflow >>>> repository. As far as I know, Github does not allow us to grant write-only >>>> permissions to the selected branch. >>>> >>>> We can keep these branches in forks managed by stakeholders teams, but >>>> I am afraid of the benefit that it will be then copied by us to our >>>> repository and then released by us. If the release was prepared by an >>>> external team, I think we should make it clear that it was prepared by >>>> another team, including by publishing on the Pypi account of the team that >>>> dealt with it. >>>> >>>> I think that everything Apache PMC releases should be prepared and >>>> created fully within the apache / airflow repositories. If stakeholders >>>> team do not have such a possibility, we should figure out that these teams >>>> become part of the community, and therefore work together with the entire >>>> community, not in isolation. Only then will we be able to act in accordance >>>> with the Apache Way <https://www.apache.org/theapacheway/>, in >>>> particular each individual person will be able to contribute to the >>>> community as an individual, and not as a company or stakeholders team >>>> (Community of Peers) and no person will get special privileges just on the >>>> basis of their employment status (Earned Authority) >>>> >>>> >>>> >>>> pon., 20 cze 2022 o 15:54 Jarek Potiuk <[email protected]> napisał(a): >>>> >>>>> > Will the people who maintain the providers' packages have the >>>>> commiter >>>>> >>>>> Nope - similarly as we do in v2-2* or what we did in 1.10.* >>>>> cherry-picking can be done in separate branches (and in this case in >>>>> forks). >>>>> Then the branch can be fast-forwarded by the committer in the >>>>> "airflow" repo. No problem with that. >>>>> >>>>> J. >>>>> >>>>> On Mon, Jun 20, 2022 at 2:53 PM Kamil Breguła <[email protected]> >>>>> wrote: >>>>> >>>>>> Will the people who maintain the providers' packages have the commiter >>>>>> status? I guess it is necessary for people to have write access to the >>>>>> repository and therefore to be able to make cherry-pick changes to the >>>>>> branch. >>>>>> >>>>>> pon., 20 cze 2022 o 09:13 Elad Kalif <[email protected]> napisał(a): >>>>>> > >>>>>> > +1 >>>>>> > From my side the proposal handles all concerns I raised in previous >>>>>> threads. >>>>>> > I think mixed-governance is a step in the right direction. >>>>>> > >>>>>> > On Wed, Jun 15, 2022 at 1:12 AM Jarek Potiuk <[email protected]> >>>>>> wrote: >>>>>> >> >>>>>> >> Hello everyone, >>>>>> >> >>>>>> >> This is a follow-up after a few discussions started about >>>>>> providers that were put on hold around the summit. I held a number of >>>>>> discussions during theSummit and after, and as result I think I have a >>>>>> proposal that can move forward some of the "stalled" decisions we need to >>>>>> make. >>>>>> >> >>>>>> >> TL;DR; >>>>>> >> >>>>>> >> My proposal is a "mixed-governance" model where "stakeholders" are >>>>>> more responsible for cherry-picking and testing their providers >>>>>> (including >>>>>> system testing) while Airflow PMC members will continue to be responsible >>>>>> for releasing them. >>>>>> >> >>>>>> >> Why do we need that? >>>>>> >> >>>>>> >> Google, Amazon and possibly others teams who are interested in >>>>>> maintaining more backwards compatible versions of their providers will >>>>>> commit to make PRs of the cherry-picks for older release branches of >>>>>> their >>>>>> providers. Those providers we release in parallel with the latest >>>>>> versions >>>>>> during the normal provider cycle. We can deprecate changes more >>>>>> aggressively in the "latest" release if we do that. >>>>>> >> >>>>>> >> Those cherry-picked PRs will be driven, tested and performed by >>>>>> the stakeholder teams (Google/Amazon, Databricks, others) and will only >>>>>> contain cherry-picks, while we - as PMC - will release them following the >>>>>> ASF rules (this is very important for the ASF to follow strict release >>>>>> policies regarding who and how performs releases). >>>>>> >> >>>>>> >> This also allows us to introduce similar rules for new provider's >>>>>> acceptance for new providers for "main releases". It also allows running >>>>>> the "system tests" for the provider under control of the stakeholder >>>>>> (after >>>>>> applying AIP-47 changes). >>>>>> >> >>>>>> >> Example 1: Google team can cherry-pick changes to a >>>>>> google-provider-6 branch and then we release a google 6.8.1 or 6.9.0 >>>>>> provider with some of the bug-fixes and features (together with - say >>>>>> latest 8.1.0). >>>>>> >> >>>>>> >> Example 2: DataLake provider from Databricks - can get accepted if >>>>>> Databricks commits to maintaining it. We will release the provider as >>>>>> long >>>>>> as Databricks maintains it. >>>>>> >> >>>>>> >> Longer context: >>>>>> >> >>>>>> >> I have - in my mind so far - a longer roadmap for providers that >>>>>> will lead them to be separated from the core and I want to write an AIP >>>>>> about that soon. This AIP will detail all the steps needed - I will work >>>>>> with multiple interested parties on it and it will take some time to >>>>>> agree >>>>>> and complete. But I want to start with something tangible that will >>>>>> solve >>>>>> quite a few problems that were raised recently and something that seems >>>>>> to >>>>>> be possible to be solved in the current provider release cycle (till the >>>>>> end of June) and test some of the governance approach. >>>>>> >> >>>>>> >> This proposal simply builds on our semver approach - we do not >>>>>> change it, we just start releasing some providers (those that have some >>>>>> backing from stakeholders) in more than one version - including "latest" >>>>>> and earlier, more backwards-compatible branches. Not all providers - just >>>>>> some. Not all branches - just those that the stakeholders will commit to >>>>>> maintain. >>>>>> >> >>>>>> >> We need such a commitment from stakeholders, because we - as the >>>>>> Airflow community and maintainers, want to only actively maintain the >>>>>> latest releases, where it is in the interest of the stakeholders to >>>>>> cherry-pick and test also earlier, more backwards compatible releases of >>>>>> their choice. >>>>>> >> >>>>>> >> What problems this proposal solves: >>>>>> >> >>>>>> >> * Problem 1: DEPRECATION REMOVAL >>>>>> >> >>>>>> >> we can remove deprecations faster in "main" versions of the >>>>>> providers - no need to introduce a deprecation policy - the stakeholders >>>>>> for the providers will take care about cherry-picking and maintaining >>>>>> more >>>>>> "backwards-compatible" versions. We are free to remove deprecations in >>>>>> major releases (in a cherry-pickable way of course). >>>>>> >> >>>>>> >> * Problem 2: PROVIDERS DIVERGENCE >>>>>> >> >>>>>> >> We avoid the problem (already happened with the composer release) >>>>>> that the stakeholders in a given provider had to release their own >>>>>> version >>>>>> which was not available in their community - with some cherry-picks. We >>>>>> want to avoid "diverging" there - by releasing the cherry-picked >>>>>> providers >>>>>> by the community, we also give other users an opportunity to follow >>>>>> "slower" deprecation policies for as long as it is maintained. >>>>>> >> >>>>>> >> * Problem 3: PROVIDERS GOVERNANCE MODEL >>>>>> >> >>>>>> >> We are going to test a governance model that we might apply when >>>>>> we split providers. We are talking about it for quite some time - but >>>>>> this >>>>>> is what helps us to test the model where stakeholders provide more >>>>>> "maintenance" while the community still takes care about releases. We (as >>>>>> community) can commit to releasing such a version of a provider as long >>>>>> as >>>>>> the stakeholder will actively maintain it. We can stop at any moment if >>>>>> we >>>>>> do not have support from the stakeholder. If it works - we can keep it >>>>>> as a >>>>>> long-term solution. In the future we can think of other scenarios >>>>>> (passing >>>>>> ownership of a provider to stakeholders who want it - providing we want >>>>>> it >>>>>> too) but we can decide about it when we learn from the mixed-governance >>>>>> model and see if it works. >>>>>> >> >>>>>> >> * Problem 4: ACCEPTING NEW PROVIDERS >>>>>> >> >>>>>> >> If this is an acceptable approach - we can also apply a very >>>>>> similar governance model to adding new providers and that should unblock >>>>>> some of the PRs that are waiting for our decision. Knowing that we are >>>>>> going to split and that we can expect "commitment" from a stakeholder, we >>>>>> should be able to accept new providers. This might be possible assuming >>>>>> that the stakeholder will make a similar commitment - but for new >>>>>> providers, that commitment might also have to cover reviewing and testing >>>>>> new changes. We might also decide as a community to stop releasing new >>>>>> providers there if such support is missing. This way we can set the >>>>>> expectations we have as a community for new providers - we will release >>>>>> them as long as the stakeholder will actively make sure it is maintained. >>>>>> >> >>>>>> >> * Problem 5. SPLITTING PROVIDERS FROM CORE >>>>>> >> >>>>>> >> We all know we want to split providers from core. By introducing >>>>>> mixed-governance we can test if it will work for the providers before we >>>>>> split them. It will take some time (and detailed AIP) to split, but in >>>>>> the >>>>>> meantime we can see if we will be able to apply the mixed-governance >>>>>> after >>>>>> the split. We will see if we can agree when it comes to expectations and >>>>>> find solutions before we actually split. >>>>>> >> >>>>>> >> J. >>>>>> >>>>>
