Hi Jarek and Airflow community, First of all, thanks a lot for your detailed explanation.
We will take it internally with our team, and feedback the conclusions. Our next step is to provide in this email chain with the list of requirements required by Airflow (after investigating all sources and links provided). We do please request the community to help us confirm, and add any missing point, to better allocate resources. Thanks a lot! Best regards David Sanchez Plaza Huawei Cloud Business Dept, International Cloud & AI David Sanchez Plaza - 大卫 Huawei HCIE Cloud Service Solutions Architect (Link<https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>) Mobile: +86 17722639223 D District, Huawei Bantian Base, Huawei, Shenzhen, China. [cid:[email protected]] [cid:[email protected]] <http://intl.huaweicloud.com/> [cid:[email protected]] <https://www.linkedin.com/showcase/huawei-cloud/> [cid:[email protected]] <https://www.facebook.com/Huawei-Cloud-1995859487299836/> [cid:[email protected]] <https://twitter.com/Huawei_Cloud> [494f24f47d32af5819074491a59e2458] <https://mp.weixin.qq.com/s?__biz=MzI1Mzc1MzMyOQ==&mid=100007026&idx=1&sn=91c07c2a030ab5fee15f24d601da530c&chksm=69cefeec5eb977faae9c29647b7512ec7e0f9bb6301e863c892dde95106d2ec732ff43662666#rd> [5edacfeb19fb3f9d99604eeaee3be363] <https://weibo.com/p/1006061930559805/home?from=page_100606&mod=TAB&is_all=1#place> From: Jarek Potiuk [mailto:[email protected]] Sent: Monday, March 20, 2023 5:13 PM To: David Sanchez Plaza <[email protected]> Cc: [email protected]; Emincan Ozcan (A) <[email protected]> Subject: Re: 【Airflow】New provider - Huawei Cloud Hello David. I think you should first review the discussions we kept on having for at least the last half a year about accepting donations of "service" bound providers from the teams of those services. General consensus is that we are extremely reluctant to get new providers added as part of the community providers and our strong preference is that service providers simply release their own providers and have the providers continue to be managed, tested and released independently by the respective teams that manage the services. You (and Huawei Cloud Provider) actually have been part of that discussion already https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 There are already a number of such providers released outside of the airflow's community repository and we are happy to add such providers to our "ecosystem page" https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers. There are multiple reasons why this is better for both - the community and service teams, all of them documented in numerous discussions - so I will not repeat them here. * https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy - Weaviate * https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. - SAS * https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 - the discussion about your provider * https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 - Pandera * https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm - Cloudera In short - adding a new provider to the community for a service that has capability of managing, hosting and releasing it on their own, puts unnecessary burden and responsibility of the maintenance and fixing bugs on the community where we often have no means, ways of testing those changes and making sure that the provider still works while the service evolves. The Huawei Cloud team is in a much better position to maintain such a provider and make sure it works than anyone here in the community. And we treat such a code as a liability, not an asset, so there must be a really good reason why we would accept it. There is absolutely nothing that Airflow provider released by Huawei cannot do compared to a community-managed provider so there is absolutely no reason why you should shy away from releasing it and hosting on your own (Similarly as Grafana, SAS, Cloudera. Pandera, Great Expectations and many others did). There are even registries of Airflow Provider (most complete is the There are a few exceptions historically where we maintain other clouds, but we are working very closely with the cloud provider teams to get on the mixed-governance process (as described in https://github.com/apache/airflow#release-process-for-providers) where all the end-to-end testing efforts remain on the side of the service provider and it's responsibility of the service provider to add a complete set of "system tests" to the provider and to run them in their cloud and provide the dashboard that can drive our release process: Example AWS dashboard here: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html IMHO - having such a dashboard and dedicated team from the cloud provider to manage the system tests in the provider and make sure the tests are running daily and take care of their health is an absolute non-negotiable prerequisite (and it should be already running and proven before we accept such provider to the community). And there should be a very good reason why we decide to accept it even if there is such a dashboard. To be perfectly honest, it's very unlikely IMHO that we will choose the path of accepting your provider rather than simply letting you manage, host and release the provider on your own. So I would encourage you not to spend too much time on trying to convince us to accept it. If you have no system tests and similar dashboard running for quite some time, the chances for it are really "None". J. On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza <[email protected]<mailto:[email protected]>> wrote: Dear community of Airflow, I am happy to announce that our development team has finished the developments of the providers to connect to Huawei Cloud Services. Therefore, we will open a PR very soon, and as a gesture of respect, we want to request for authorization from community. Our team is reviewing the guidelines to comply with the requirements. We will continue to update and add more providers in the near future. We hope this integration can grow customer base and Airflow project. Best regards David Sanchez Plaza Huawei Cloud Business Dept, International Cloud & AI David Sanchez Plaza - 大卫 Huawei HCIE Cloud Service Solutions Architect (Link<https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>) Mobile: +86 17722639223 D District, Huawei Bantian Base, Huawei, Shenzhen, China. <http://intl.huaweicloud.com/>
