kaxil commented on a change in pull request #11529:
URL: https://github.com/apache/airflow/pull/11529#discussion_r504930868
##########
File path: CONTRIBUTING.rst
##########
@@ -520,9 +524,94 @@ yandexcloud, all, devel_ci
.. END EXTRAS HERE
+Provider packages
+-----------------
-Airflow dependencies
---------------------
+Airflow 2.0 is split into core and providers. They are delivered as separate
packages:
+
+* apache-airflow: core
+* apache-airflow-providers-*: More than 50 provider packages
+
+In Airflow 1.10 all those providers were installed together within one single
package and when you installed
+airflow locally, from sources, they were also installed. In Airflow 2.0,
providers are separated out,
+and not installed together with the core, unless you set
INSTALL_PROVIDERS_FROM_SOURCES environment
+variable to ``true``.
+
+In Breeze - which is a development variable, INSTALL_PROVIDERS_FROM_SOURCES is
set to true, but you
+can also add ``--skip-installing-airflow-providers`` flag to Breeze to skip
installing them when
+building the images.
+
+One watch-out - providers are still always installed (or rather available) if
you install airflow from
+sources using ``-e`` (or ``--editable``) flag. In such case airflow is read
directly from the sources
+without copying airflow packages to the usual installation location, and since
'providers' folder is
+in this airflow folder - the providers package is importable.
+
+Some of the packages have cross-dependencies with other providers packages.
This typically happens for
+transfer operators where operators use hooks from the other providers in case
they are transferring
+data between the providers. The list of dependencies is maintained
(automatically with pre-commits)
+in the ``airflow/providers/dependencies.json``. Pre-commits are also used to
generate dependencies.
+The dependency list is automatically used during pypi packages generation.
+
+Cross-dependencies between provider packages are converted into extras - if
you need functionality from
+the other provider package you can install it adding [extra] after the
+apache-airflow-backport-providers-PROVIDER for example ``pip install
+apache-airflow-backport-providers-google[amazon]`` in case you want to use GCP
+transfer operators from Amazon ECS.
+
+If you add a new dependency between different providers packages, it will be
detected automatically during
+pre-commit phase and pre-commit will fail - and add entry in dependencies.json
so that the package extra
+dependencies are properly added when package is installed.
+
+You can regenerate the whole list of provider dependencies by running this
command (you need to have
+``pre-commits`` installed).
+
+.. code-block:: bash
+
+ pre-commit run build-providers-dependencies
+
+
+Here is the list of packages and their extras:
+
+
+ .. START PACKAGE DEPENDENCIES HERE
+
+========================== ===========================
+Package Extras
+========================== ===========================
+amazon apache.hive,google,imap,mongo,mysql,postgres,ssh
+apache.druid apache.hive
+apache.hive amazon,microsoft.mssql,mysql,presto,samba,vertica
+apache.livy http
+dingding http
+discord http
+google
amazon,apache.cassandra,cncf.kubernetes,facebook,microsoft.azure,microsoft.mssql,mysql,postgres,presto,sftp
+hashicorp google
+microsoft.azure google,oracle
+microsoft.mssql odbc
+mysql amazon,presto,vertica
+opsgenie http
+postgres amazon
+sftp ssh
+slack http
+snowflake slack
+========================== ===========================
+
+ .. END PACKAGE DEPENDENCIES HERE
+
+Backport providers
+------------------
+
+You can also build backport provider packages for Airflow 1.10. They aim to
provide a bridge when users
+of Airflow 1.10 want to migrate to Airflow 2.0. The backport packages are
named similarly to the
+provider packages, but with "backport" added:
+
+* 'apache-airflow-backport-provider-*
Review comment:
```suggestion
* 'apache-airflow-backport-provider-*'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
