eladkal commented on code in PR #30657: URL: https://github.com/apache/airflow/pull/30657#discussion_r1167481923
########## PROVIDERS.rst: ########## @@ -0,0 +1,243 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +************************ +Apache Airflow Providers +************************ + +.. contents:: :local: + +What is a provider? +=================== + +Airflow 2.0 introduced the concept of providers. Providers are packages that contain integrations with +external systems. They are meant to extend capabilities of the core "Apache Airflow". Thus they are +part of the vision of Airflow-as-a-Platform - where the Airflow Core provides basic data-workflow scheduling +and management capabilities and can be extended by implementing Open APIs Airflow supports, adding +Plugins that can add new features to the Core, and adding Providers that allow to interact with external +systems. + +The providers are released separately from the core Airflow and they are versioned independently. The +w,ays how providers can extend the Airflow Core, types of providers, can be found at the +`Providers page <https://airflow.apache.org/docs/apache-airflow-providers/index.html>`_. You can also find +out there how you can create your own provider. + +Providers can be maintainers and released by the Airflow community or by 3rd-party teams. In any event they +are released independently of the Airflow Core package. + +When we release Airflow Core, it is released with constraints, that are using the latest released version +of providers, and our published convenience images contain a subset of most popular community providers, +however our users are free to upgrade and downgrade providers independently of the Airflow Core version +as they see fit, as long as it does not cause conflicting dependencies. + +You can read more about it in the +`Installation and upgrade scenarios <https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios>`_ +chapter of our user documentation. + +Community managed providers +=========================== + +When providers are accepted by the community, the providers must follow the Apache Software Foundation +rules and policies when it comes to accepting contributions and releasing new versions of the providers. This +means that the code changes in the providers must be reviewed by Airflow committers and merged when they +are accepted by them, also they must implement all the test and documentation requirements that are +required by the community. + +They providers in their latest version in "main" branch of airflow repository, are also tested together +with other community providers and one of the key properties of the community providers is that the latest +version of providers contribute their dependencies to constraints of Airflow published when Airflow Core is +released - which means that when you are using constraints published by Airflow, you can install all +the providers together and they are tested together. This allows to add an optional "extra" to Airflow +for each provider, so that the providers can be installed together with Airflow by specifying the +"extra" in the installation command. + +Because of the constraint and potential conflicting dependencies, the community providers have to be regularly +updated and the community might decide to suspend releases of a provider if we find out that we have trouble +with updating the dependencies of the provider, or if we find out that the provider is not compatible with +other providers. See the section below for more details on suspending releases of the community providers. + +List of all available community providers is available at the `Providers index <https://airflow.apache.org/docs/>`_. + +Accepting new community providers +================================= + +Accepting new community providers should be a deliberate process that requires ``[DISCUSSION]`` +followed by ``[VOTE]`` thread at the airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_. +In case of provider is integrating with an open-source software rather than service (particularly if the +open-source software is an Apache Software Foundation, Linux Software Foundation or similar organisation with +well established governance processes that are not strictly vendor-controlled) and when the software is +well established an popular it might be enough to have a good and complete PR of the provider, ideally +with a good test coverage (including integration tests) and documentation approved by the team +``[LAZY CONSENSUS]`` on the devlist might be enough to accept the provider. Review Comment: > I think when getting new PR in an existing provider we have, we have much better knowledge - will there be maintenance, will there be support in case of problem, what happens when there is a need to upgrade or when the service gets disabled. We are going through that with Google team and while a bit late, the make a huge effort to make their provider updated now. This is a good lesson for us. I'm not referring to the case that we have now. I'm referring for a case that adding a new extension for a service that is not part of the original provider scope. If tomorrow Google acquire a company technically it allows them to add a new integration for it without violating this policy. I don't mean to cases where under google.cloud we add more services and improve the support there i'm referring to cases of adding a whole new service branch like: leveldb, looker etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
