Hello David, In case you were not following: The important part of the discussion and reasoning (and strong preference of external providers) have now been captured in our repo https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers. You can refer to it when you further think about submitting your provider, the whole document is intended so that we do not have to discuss it again, next time when someone will ask about it.
Nothing changed, but things got explained and clarified also with reasoning why this is the case. J. On Tue, Mar 21, 2023 at 1:48 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Just to be brutally honest your resources are better allocated to > release and maintain the provider on your own. > > It's not only requirements but you would have to justify why > maintenance burden of Huawei Cloud provider is supposed to be done by > the community and not Huawei Cloud team (where it would normally > belong). > I can't see the reason why we would want it in the community to > maintain and until there is a good justification, my vote will be -1 > personally. > > It's my personal opinion of course and ultimately not the requirements > matter but formally a vote would be needed following > https://www.apache.org/foundation/voting.html#votes-on-code-modification. > You might want to convince me and others otherwise, but I seriously > doubt you will get it passed the vote if you call for it and there is > no good justification for it. I think spending time on releasing it as > your provider is a much better spent time for you. > > J. > > On Tue, Mar 21, 2023 at 11:07 AM David Sanchez Plaza > <david.sanchez.pl...@huawei.com> wrote: > > > > Hi Jarek and Airflow community, > > > > > > > > First of all, thanks a lot for your detailed explanation. > > > > > > > > We will take it internally with our team, and feedback the conclusions. > > > > > > > > Our next step is to provide in this email chain with the list of > requirements required by Airflow (after investigating all sources and links > provided). We do please request the community to help us confirm, and add > any missing point, to better allocate resources. > > > > > > > > > > > > Thanks a lot! > > > > > > > > Best regards > > > > David Sanchez Plaza > > > > > > > > Huawei Cloud Business Dept, International Cloud & AI > > > > David Sanchez Plaza - 大卫 > > > > Huawei HCIE Cloud Service Solutions Architect (Link) > > > > Mobile: +86 17722639223 > > > > D District, Huawei Bantian Base, Huawei, Shenzhen, China. > > > > > > > > > > > > From: Jarek Potiuk [mailto:ja...@potiuk.com] > > Sent: Monday, March 20, 2023 5:13 PM > > To: David Sanchez Plaza <david.sanchez.pl...@huawei.com> > > Cc: dev@airflow.apache.org; Emincan Ozcan (A) < > emincan.ozc...@huawei-partners.com> > > Subject: Re: 【Airflow】New provider - Huawei Cloud > > > > > > > > Hello David. > > > > > > > > I think you should first review the discussions we kept on having for at > least the last half a year about accepting donations of "service" bound > providers from the teams of those services. General consensus is that we > are extremely reluctant to get new providers added as part of the community > providers and our strong preference is that service providers simply > release their own providers and have the providers continue to be managed, > tested and released independently by the respective teams that manage the > services. > > > > > > > > You (and Huawei Cloud Provider) actually have been part of that > discussion already > https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 > > > > > > > > There are already a number of such providers released outside of the > airflow's community repository and we are happy to add such providers to > our "ecosystem page" > https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers > . > > > > > > > > There are multiple reasons why this is better for both - the community > and service teams, all of them documented in numerous discussions - so I > will not repeat them here. > > > > > > > > * https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy - > Weaviate > > > > * https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. - > SAS > > * https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 - > the discussion about your provider > > > > * https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 - > Pandera > > * https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm - > Cloudera > > > > > > > > In short - adding a new provider to the community for a service that has > capability of managing, hosting and releasing it on their own, puts > unnecessary burden and responsibility of the maintenance and fixing bugs on > the community where we often have no means, ways of testing those changes > and making sure that the provider still works while the service evolves. > The Huawei Cloud team is in a much better position to maintain such a > provider and make sure it works than anyone here in the community. And we > treat such a code as a liability, not an asset, so there must be a really > good reason why we would accept it. There is absolutely nothing that > Airflow provider released by Huawei cannot do compared to a > community-managed provider so there is absolutely no reason why you should > shy away from releasing it and hosting on your own (Similarly as Grafana, > SAS, Cloudera. Pandera, Great Expectations and many others did). There are > even registries of Airflow Provider (most complete is the > > > > > > > > There are a few exceptions historically where we maintain other clouds, > but we are working very closely with the cloud provider teams to get on the > mixed-governance process (as described in > https://github.com/apache/airflow#release-process-for-providers) where > all the end-to-end testing efforts remain on the side of the service > provider and it's responsibility of the service provider to add a complete > set of "system tests" to the provider and to run them in their cloud and > provide the dashboard that can drive our release process: > > > > > > > > Example AWS dashboard here: > https://aws-mwaa.github.io/open-source/system-tests/dashboard.html > > > > > > > > IMHO - having such a dashboard and dedicated team from the cloud > provider to manage the system tests in the provider and make sure the tests > are running daily and take care of their health is an absolute > non-negotiable prerequisite (and it should be already running and proven > before we accept such provider to the community). And there should be a > very good reason why we decide to accept it even if there is such a > dashboard. > > > > > > > > To be perfectly honest, it's very unlikely IMHO that we will choose the > path of accepting your provider rather than simply letting you manage, host > and release the provider on your own. So I would encourage you not to > spend too much time on trying to convince us to accept it. If you have no > system tests and similar dashboard running for quite some time, the chances > for it are really "None". > > > > > > > > J. > > > > > > > > > > > > On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza > <david.sanchez.pl...@huawei.com.invalid> wrote: > > > > Dear community of Airflow, > > > > > > > > I am happy to announce that our development team has finished the > developments of the providers to connect to Huawei Cloud Services. > Therefore, we will open a PR very soon, and as a gesture of respect, we > want to request for authorization from community. > > > > > > > > Our team is reviewing the guidelines to comply with the requirements. > > > > > > > > We will continue to update and add more providers in the near future. We > hope this integration can grow customer base and Airflow project. > > > > > > > > Best regards > > > > David Sanchez Plaza > > > > > > > > Huawei Cloud Business Dept, International Cloud & AI > > > > David Sanchez Plaza - 大卫 > > > > Huawei HCIE Cloud Service Solutions Architect (Link) > > > > Mobile: +86 17722639223 > > > > D District, Huawei Bantian Base, Huawei, Shenzhen, China. > > > > > > > > > > > > >
