jscheffl commented on code in PR #36537:
URL: https://github.com/apache/airflow/pull/36537#discussion_r1443481860
##########
CONTRIBUTING.rst:
##########
@@ -841,29 +841,106 @@ Extras
------
There are a number of extras that can be specified when installing Airflow.
Those
-extras can be specified after the usual pip install - for example
-``pip install -e .[ssh]``. For development purpose there is a ``devel`` extra
that
-installs all development dependencies. There is also ``devel_ci`` that installs
-all dependencies needed in the CI environment.
+extras can be specified after the usual pip install - for example ``pip
install -e.[editable_ssh]`` for editable
Review Comment:
I assume at least a space is missing and I am not sure whether the name is
correct? Is it really `editable` or `devel`?
```suggestion
extras can be specified after the usual pip install - for example ``pip
install -e .[editable_ssh]`` for editable
```
```suggestion
extras can be specified after the usual pip install - for example ``pip
install -e .[devel_ssh]`` for editable
```
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -97,7 +106,7 @@ with a consistent set of dependencies based on constraint
files provided by Airf
.. code-block:: bash
:substitutions:
- pip install apache-airflow[google,amazon,apache.spark]==|version| \
+ pip install apache-airflow[google,amazon,apache-spark]==|version| \
Review Comment:
Inconsistent or typo? Below you are consistent with `_`
```suggestion
pip install apache-airflow[google,amazon,apache_spark]==|version| \
```
##########
airflow/providers/fab/provider.yaml:
##########
@@ -36,6 +36,11 @@ versions:
dependencies:
- apache-airflow>=2.9.0
- flask>=2.2,<2.3
+ # We are tightly coupled with FAB version as we vendored-in part of FAB code
related to security manager
+ # This is done as part of preparation to removing FAB as dependency, but we
are not ready for it yet
+ # Every time we update FAB version here, please make sure that you review
the classes and models in
+ # `airflow/providers/fan/auth_manager/security_manager/override.py` with
their upstream counterparts.
Review Comment:
Maybe this note does not belong to this PR and I lag a bit of history and
when changing/fixing something in FAB last time I was wondering why actually we
have a partly fork of the same code - I assume this is for historic reasons.
I'd rather favor to really clearly separate the files or contribute upstream
the diff... but this would be also outside of this PR actually...
##########
airflow/providers/fab/provider.yaml:
##########
@@ -36,6 +36,11 @@ versions:
dependencies:
- apache-airflow>=2.9.0
- flask>=2.2,<2.3
+ # We are tightly coupled with FAB version as we vendored-in part of FAB code
related to security manager
+ # This is done as part of preparation to removing FAB as dependency, but we
are not ready for it yet
+ # Every time we update FAB version here, please make sure that you review
the classes and models in
+ # `airflow/providers/fan/auth_manager/security_manager/override.py` with
their upstream counterparts.
Review Comment:
typo
```suggestion
# `airflow/providers/fab/auth_manager/security_manager/override.py` with
their upstream counterparts.
```
##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########
@@ -53,8 +53,8 @@
LONG_PROVIDERS_PREFIX = "apache-airflow-providers-"
-# TODO: use single source of truth for those
-# for now we need to keep them in sync with the ones in setup.py
+# TODO(potiuk): use single source of truth for those
Review Comment:
Would this be also a candidate for a separate PR? Instead of putting your
name on it, is there an existing issue to point to?
##########
INSTALL:
##########
@@ -1,118 +1,324 @@
# INSTALL / BUILD instructions for Apache Airflow
-This is a generic installation method that requires a number of dependencies
to be installed.
+## Basic installation of Airflow from sources and development environment
setup
+
+This is a generic installation method that requires minimum starndard tools to
develop airflow and
+test it in local virtual environment (using standard CPyhon installation and
`pip`).
Depending on your system you might need different prerequisites, but the
following
systems/prerequisites are known to work:
-Linux (Debian Bullseye, Bookworm and Linux Mint Debbie):
+Linux (Debian Bookworm):
-sudo apt install build-essential python3-dev libsqlite3-dev openssl \
- sqlite default-libmysqlclient-dev libmysqlclient-dev
postgresql
+ sudo apt install -y --no-install-recommends apt-transport-https apt-utils
ca-certificates \
+ curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
+ ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales libffi8
libldap-2.5-0 libssl3 netcat-openbsd \
+ lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo
unixodbc
-On Ubuntu 20.04 you may get an error of mariadb_config not found
-and mysql_config not found.
+You might need to install MariaDB development headers to build some of the
dependencies
-Install MariaDB development headers:
-sudo apt-get install libmariadb-dev libmariadbclient-dev
+ sudo apt-get install libmariadb-dev libmariadbclient-dev
-MacOS (Mojave/Catalina):
+MacOS (Mojave/Catalina) you might need to to install XCode command line tools
and brew and those packages:
-brew install sqlite mysql postgresql
+ brew install sqlite mysql postgresql
-# [required] fetch the tarball and untar the source move into the directory
that was untarred.
+## Downloading and installing Airflow from sources
-# [optional] run Apache RAT (release audit tool) to validate license headers
-# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat
-java -jar apache-rat.jar -E ./.rat-excludes -d .
+While you can get Airflow sources in various ways (including cloning
https://github.com/apache/airflow/), the
+canonical way to download it is to fetch the tarball published at
https://downloads.apache.org where you can
+also verify checksum, signatures of the downloaded file. You can then and
un-tar the source move into the
+directory that was un-tarred.
-# [optional] Airflow pulls in quite a lot of dependencies in order
-# to connect to other services. You might want to test or run Airflow
-# from a virtual env to make sure those dependencies are separated
-# from your system wide versions
+When you download source packages from https://downloads.apache.org, you
download sources of Airflow and
+all providers separately, however when you clone the GitHub repository at
https://github.com/apache/airflow/
+you get all sources in one place. This is the most convenient way to develop
Airflow and Providers together.
+otherwise you have to separately install Airflow and Providers from sources in
the same environment, which
+is not as convenient.
-python3 -m venv PATH_TO_YOUR_VENV
-source PATH_TO_YOUR_VENV/bin/activate
+## Creating virtualenv
-# [required] building and installing by pip (preferred)
-pip install .
+Airflow pulls in quite a lot of dependencies in order to connect to other
services. You generally want to
+test or run Airflow from a virtual env to make sure those dependencies are
separated from your system
+wide versions. Using system-installed Python installation is strongly
discouraged as the versions of Python
+shipped with operating system often have a number of limitations and are not
up to date. It is recommended
+to install Python using either https://www.python.org/downloads/ or other
tools that use them. See later
+for description of `Hatch` as one of the tools that is Airflow's tool of
choice to build Airflow packages.
-# or directly
-python setup.py install
+Once you have a suitable Python version installed, you can create a virtualenv
and activate it:
-# You can also install recommended version of the dependencies by using
-# constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as constraint file.
This is needed in case
-# you have problems with installing the current requirements from PyPI.
-# There are different constraint files for different python versions. For
example"
+ python3 -m venv PATH_TO_YOUR_VENV
+ source PATH_TO_YOUR_VENV/bin/activate
-pip install . \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
+## Installing airflow locally
+Installing airflow locally can be done using pip - note that this will install
"development" version of
+Airflow, where all providers are installed from local sources (if they are
available), not from `pypi`.
+It will also not include pre-installed providers installed from PyPI. In case
you install from sources of
+just Airflow, you need to install separately each provider that you want to
develop. In case you install
+from GitHub repository, all the current providers are available after
installing Airflow.
-By default `pip install` in Airflow 2.0 installs only the provider packages
that are needed by the extras and
-install them as packages from PyPI rather than from local sources:
+ pip install .
-pip install .[google,amazon] \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
+If you develop Airflow and iterate on it you should install it in editable
mode (with -e) flag and then
+you do not need to re-install it after each change to sources. This is useful
if you want to develop and
+iterate on Airflow and Providers (together) if you install sources from cloned
GitHub repository.
+ pip install -e .
-You can upgrade just airflow, without paying attention to provider's
dependencies by using 'constraints-no-providers'
-constraint files. This allows you to keep installed provider packages.
-pip install . --upgrade \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt";
+You can also install optional packages that are needed to run certain tests.
In case of local installation
+for example you can install all prerequisites for google provider, tests and
+all hadoop providers with this command:
+
+ pip install -e ".[editable_google,devel_tests,devel_hadoop]"
Review Comment:
Maybe also corresponding to comment above. I am now confused between
`editable_` and `devel_` prefix. Maybe two additional sentences here would be
good.
##########
dev/old_setup/MANIFEST.in:
##########
Review Comment:
Why keeping this file in a subfolder?
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
Review Comment:
Maybe not really/fully relating to this PR but just "stumbling" over this:
What is "All database" referring to? Is there a list of "all"? Referring to
provider packages "smelling" like DB or all database types we support in
Airflow for the Airflow meta data (a.k.a sqlite, postgres, mysql/maria nowadays)
##########
dev/breeze/src/airflow_breeze/utils/path_utils.py:
##########
@@ -97,7 +97,7 @@ def get_package_setup_metadata_hash() -> str:
return "NOT FOUND"
-def get_sources_setup_metadata_hash(sources: Path) -> str:
+def get_pyproject_toml_hash(sources: Path) -> str:
try:
the_hash = hashlib.new("blake2b")
the_hash.update((sources / "dev" / "breeze" /
"pyproject.toml").read_bytes())
Review Comment:
Maybe missing context for me but why is the hash generated on a different
TOML than the project? Is this for the setup of breeze itself? If it is for
breeze itself we should rename the function to make clear it is the breeze
hash. Otherwise you could use the beautiful new constant from above.
```suggestion
the_hash.update((sources / "dev" / "breeze" /
PYPROJECT_TOML).read_bytes())
```
##########
LOCAL_VIRTUALENV.rst:
##########
@@ -169,38 +171,31 @@ for different python versions). For development on
current main source:
.. code-block:: bash
# use the same version of python as you are working with, 3.8, 3.9, 3.10
or 3.11
- pip install -e ".[devel,<OTHER EXTRAS>]" \
+ pip install -e ".[devel,devel_tests,editable_amazon,...<OTHER
DEVEL_EXTRAS>]" \
--constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt";
This will install Airflow in 'editable' mode - where sources of Airflow are
taken directly from the source
-code rather than moved to the installation directory. During the installation
airflow will install - but then
-automatically remove all provider packages installed from PyPI - instead it
will automatically use the
-provider packages available in your local sources.
+code rather than moved to the installation directory.
+
+TODO(potiuk): Clarify preinstalled
You can also install Airflow in non-editable mode:
.. code-block:: bash
# use the same version of python as you are working with, 3.8, 3.9, 3.10
or 3.11
- pip install ".[devel,<OTHER EXTRAS>]" \
+ pip install ".[devel,devel_tests,editable_amazon,...<OTHER DEVEL_EXTRAS>]"
\
Review Comment:
Copy&paste error? This was describing the non-editable way, should be like?
```suggestion
pip install ".[devel,devel_tests,amazon,google,...<OTHER DEVEL_EXTRAS>]"
\
```
##########
airflow/utils/dot_renderer.py:
##########
@@ -19,9 +19,14 @@
"""Renderer DAG (tasks and dependencies) to the graphviz object."""
from __future__ import annotations
+import warnings
from typing import TYPE_CHECKING, Any
-import graphviz
+try:
+ import graphviz
Review Comment:
Same like comment above with FAB - the changes in this file are a candidate
to separate-out to another PR
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -18,17 +18,24 @@
Reference for package extras
''''''''''''''''''''''''''''
-Here's the list of all the extra dependencies of Apache Airflow.
+Airflow has a number of optional "extras" that you can use to add features to
your installation when you
+are installing Airflow. Those extras are a good way for the users to manage
their installation, but also
+they are useful for contributors to airflow when they want to contribute some
of the featuers - including
+optional integrations of Airflow - via providers.
-The entries with ``*`` in the ``Preinstalled`` column indicate that those
extras (providers) are always
-pre-installed when Airflow is installed.
+,, warning::
Review Comment:
Typo?
```suggestion
.. warning::
```
##########
INSTALL:
##########
@@ -1,118 +1,324 @@
# INSTALL / BUILD instructions for Apache Airflow
-This is a generic installation method that requires a number of dependencies
to be installed.
+## Basic installation of Airflow from sources and development environment
setup
+
+This is a generic installation method that requires minimum starndard tools to
develop airflow and
+test it in local virtual environment (using standard CPyhon installation and
`pip`).
Depending on your system you might need different prerequisites, but the
following
systems/prerequisites are known to work:
-Linux (Debian Bullseye, Bookworm and Linux Mint Debbie):
+Linux (Debian Bookworm):
-sudo apt install build-essential python3-dev libsqlite3-dev openssl \
- sqlite default-libmysqlclient-dev libmysqlclient-dev
postgresql
+ sudo apt install -y --no-install-recommends apt-transport-https apt-utils
ca-certificates \
+ curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
+ ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales libffi8
libldap-2.5-0 libssl3 netcat-openbsd \
+ lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo
unixodbc
-On Ubuntu 20.04 you may get an error of mariadb_config not found
-and mysql_config not found.
+You might need to install MariaDB development headers to build some of the
dependencies
-Install MariaDB development headers:
-sudo apt-get install libmariadb-dev libmariadbclient-dev
+ sudo apt-get install libmariadb-dev libmariadbclient-dev
-MacOS (Mojave/Catalina):
+MacOS (Mojave/Catalina) you might need to to install XCode command line tools
and brew and those packages:
-brew install sqlite mysql postgresql
+ brew install sqlite mysql postgresql
-# [required] fetch the tarball and untar the source move into the directory
that was untarred.
+## Downloading and installing Airflow from sources
-# [optional] run Apache RAT (release audit tool) to validate license headers
-# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat
-java -jar apache-rat.jar -E ./.rat-excludes -d .
+While you can get Airflow sources in various ways (including cloning
https://github.com/apache/airflow/), the
+canonical way to download it is to fetch the tarball published at
https://downloads.apache.org where you can
+also verify checksum, signatures of the downloaded file. You can then and
un-tar the source move into the
+directory that was un-tarred.
-# [optional] Airflow pulls in quite a lot of dependencies in order
-# to connect to other services. You might want to test or run Airflow
-# from a virtual env to make sure those dependencies are separated
-# from your system wide versions
+When you download source packages from https://downloads.apache.org, you
download sources of Airflow and
+all providers separately, however when you clone the GitHub repository at
https://github.com/apache/airflow/
+you get all sources in one place. This is the most convenient way to develop
Airflow and Providers together.
+otherwise you have to separately install Airflow and Providers from sources in
the same environment, which
+is not as convenient.
-python3 -m venv PATH_TO_YOUR_VENV
-source PATH_TO_YOUR_VENV/bin/activate
+## Creating virtualenv
-# [required] building and installing by pip (preferred)
-pip install .
+Airflow pulls in quite a lot of dependencies in order to connect to other
services. You generally want to
+test or run Airflow from a virtual env to make sure those dependencies are
separated from your system
+wide versions. Using system-installed Python installation is strongly
discouraged as the versions of Python
+shipped with operating system often have a number of limitations and are not
up to date. It is recommended
+to install Python using either https://www.python.org/downloads/ or other
tools that use them. See later
+for description of `Hatch` as one of the tools that is Airflow's tool of
choice to build Airflow packages.
-# or directly
-python setup.py install
+Once you have a suitable Python version installed, you can create a virtualenv
and activate it:
-# You can also install recommended version of the dependencies by using
-# constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as constraint file.
This is needed in case
-# you have problems with installing the current requirements from PyPI.
-# There are different constraint files for different python versions. For
example"
+ python3 -m venv PATH_TO_YOUR_VENV
+ source PATH_TO_YOUR_VENV/bin/activate
-pip install . \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
+## Installing airflow locally
+Installing airflow locally can be done using pip - note that this will install
"development" version of
+Airflow, where all providers are installed from local sources (if they are
available), not from `pypi`.
+It will also not include pre-installed providers installed from PyPI. In case
you install from sources of
+just Airflow, you need to install separately each provider that you want to
develop. In case you install
+from GitHub repository, all the current providers are available after
installing Airflow.
-By default `pip install` in Airflow 2.0 installs only the provider packages
that are needed by the extras and
-install them as packages from PyPI rather than from local sources:
+ pip install .
-pip install .[google,amazon] \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
+If you develop Airflow and iterate on it you should install it in editable
mode (with -e) flag and then
+you do not need to re-install it after each change to sources. This is useful
if you want to develop and
+iterate on Airflow and Providers (together) if you install sources from cloned
GitHub repository.
+ pip install -e .
-You can upgrade just airflow, without paying attention to provider's
dependencies by using 'constraints-no-providers'
-constraint files. This allows you to keep installed provider packages.
-pip install . --upgrade \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt";
+You can also install optional packages that are needed to run certain tests.
In case of local installation
+for example you can install all prerequisites for google provider, tests and
+all hadoop providers with this command:
+
+ pip install -e ".[editable_google,devel_tests,devel_hadoop]"
+
+
+or you can install all packages needed to run tests for core airflow:
+
+ pip install -e ".[devel]"
+
+or you can install all packages needed to run tests for core, providers and
all extensions of airflow:
+
+ pip install -e ".[devel_all]"
+
+You can see the list of all available extras below.
+
+# Using Hatch to manage your Python, virtualenvs and build packages
+
+Airflow uses [hatch](https://hatch.pypa.io/) as a build and development tool
of choice. It is one of popular
+build tools and environment managers for Python, maintained by the Python
Packaging Authority.
+It is an optional tool that is only really needed when you want to build
packages from sources, but
+it is also very convenient to manage your Python versions and virtualenvs.
+
+Airflow project contains some pre-defined virtualenv definitions in
``pyproject.toml`` that can be
+easily used by hatch to create your local venvs. This is not necessary for you
to develop and test
+Airflow, but it is a convenient way to manage your local Python versions and
virtualenvs.
+
+## Installing Hatch
+
+You can install hat using various other ways (including Gui installers).
+
+Example using `pipx`:
+
+ pipx install hatch
+
+We recommend using `pipx` as you can manage installed Python apps easily and
later use it
+to upgrade `hatch` easily as needed with:
+
+ pipx upgrade hatch
+
+## Using Hatch to manage your Python versions
+
+You can also use hatch to install and manage airflow virtualenvs and
development
+environments. For example, you can install Python 3.10 with this command:
+
+ hatch python install 3.10
+
+or install all Python versions that are used in Airflow:
+
+ hatch python install all
+
+## Using Hatch to manage your virtualenvs
+
+Airflow has some pre-defined virtualenvs that you can use to develop and test
airflow.
+You can see the list of available envs with:
+
+ hatch show env
+
+This is what it shows currently:
+
+┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Name ┃ Type ┃ Dependencies ┃ Description
┃
+┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ default │ virtual │ apache-airflow[devel] │ Default environment with
Python 3.8 for maximum compatibility │
+├─────────────┼─────────┼───────────────────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-38 │ virtual │ apache-airflow[devel] │ Environment with Python 3.8
│
+├─────────────┼─────────┼───────────────────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-39 │ virtual │ apache-airflow[devel] │ Environment with Python 3.9
│
+├─────────────┼─────────┼───────────────────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-310 │ virtual │ apache-airflow[devel] │ Environment with Python 3.10
│
+├─────────────┼─────────┼───────────────────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-311 │ virtual │ apache-airflow[devel] │ Environment with Python 3.11
│
+└─────────────┴─────────┴───────────────────────┴───────────────────────────────────────────────────────────────┘
+
+The default env (if you have not used one explicitly) is `default` and it is a
Python 3.8
+virtualenv for maximum compatibility. The default extra with which the
environment is create is "devel", which
+should be enough to develop and test basic airflow core tests. You can create
the default environment with:
+
+ hatch env create
+
+You can create specific environment by using them in create command:
+
+ hatch env create airflow-310
+
+You can install extras in the environment by running pip command:
+
+ hatch -e airflow-310 run -- pip install ".[editable_google]"
+
+And you can enter the environment with running a shell of your choice (for
example zsh) where you
+can run any commands
+
+ hatch -e airflow-310 shell
+
+You can also see where hatch created the virtualenvs and use it in your IDE or
activate it manually:
+
+ hatch env find airflow-310
+
+You will get path similar to:
+
+ /Users/jarek/Library/Application
Support/hatch/env/virtual/apache-airflow/TReRdyYt/apache-airflow
+
+Then you will find `python` binary and `activate` script in the `bin`
sub-folder of this directory and
+you can configure your IDE to use this python virtualenv if you want to use
that environment in your IDE.
+
+You can also set default environment by HATCH_ENV environment variable.
+You can clean the env by running:
-You can also install airflow in "editable mode" (with -e) flag and then
provider packages are
-available directly from the sources (and the provider packages installed from
PyPI are UNINSTALLED in
-order to avoid having providers in two places. And `provider.yaml` files are
used to discover capabilities
-of the providers which are part of the airflow source code.
+ hatch env prune
-You can read more about `provider.yaml` and community-managed providers in
-https://airflow.apache.org/docs/apache-airflow-providers/index.html for
developing custom providers
-and in ``CONTRIBUTING.rst`` for developing community maintained providers.
+More information about hatch can be found in
https://hatch.pypa.io/1.9/environment/
-This is useful if you want to develop providers:
+## Using Hatch to build your packages
+
+You can use hatch to build installable package from the airflow sources. Such
package will
+include all metadata that is configured in `pyproject.toml` and will be
installable with pip.
+
+The packages will have pre-installed dependencies for providers that are always
+installed when Airflow is installed from PyPI. By default both `wheel` and
`sdist` packages are built.
+
+ hatch build
+
+You can also build only `wheel` or `sdist` packages:
+
+ hatch build -t wheel
+ hatch build -t sdist
+
+## Installing recommended version of dependencies
+
+Whatever virtualenv solution you use, when you want to make sure you are using
the same
+version of dependencies as in main, you can install recommended version of the
dependencies by using
+constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as `constraint` file.
This might be useful
+to avoid "works-for-me" syndrome, where you use different version of
dependencies than the ones
+that are used in main, CI tests and by other contributors.
+
+There are different constraint files for different python versions. For
example this command will install
+all basic devel requirements and requirements of google provider as last
successfully tested for Python 3.8:
+
+ pip install -e ".[devel,editable_google]"" \
+ --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
+
+You can upgrade just airflow, without paying attention to provider's
dependencies by using
+the 'constraints-no-providers' constraint files. This allows you to keep
installed provider dependencies
+and install to latest supported ones by pure airflow core.
pip install -e . \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
-
-You can also skip installing provider packages from PyPI by setting
INSTALL_PROVIDERS_FROM_SOURCE to "true".
-In this case Airflow will be installed in non-editable mode with all providers
installed from the sources.
-Additionally `provider.yaml` files will also be copied to providers folders
which will make the providers
-discoverable by Airflow even if they are not installed from packages in this
case.
-
-INSTALL_PROVIDERS_FROM_SOURCES="true" pip install . \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
-
-Airflow can be installed with extras to install some additional features (for
example 'async' or 'doc' or
-to install automatically providers and all dependencies needed by that
provider:
-
-pip install .[async,google,amazon] \
- --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt";
-
-The list of available extras:
-
-# START EXTRAS HERE
-aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas,
apache.beam, apache.cassandra,
-apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive,
apache.impala, apache.kafka,
-apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark,
apache.webhdfs, apprise,
-arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery,
cgroups, cloudant,
-cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog,
dbt.cloud,
-deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord,
doc, doc_gen, docker,
-druid, elasticsearch, exasol, fab, facebook, ftp, gcp, gcp_api, github,
github_enterprise, google,
-google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins,
kerberos, kubernetes,
-ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp,
microsoft.winrm, mongo, mssql,
-mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie,
oracle, otel, pagerduty,
-pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto,
rabbitmq, redis, s3, s3fs,
-salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack,
smtp, snowflake,
-spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica,
virtualenv, weaviate,
-webhdfs, winrm, yandex, zendesk
-# END EXTRAS HERE
-
-# For installing Airflow in development environments - see CONTRIBUTING.rst
-
-# COMPILING FRONT-END ASSETS (in case you see "Please make sure to build the
frontend in static/ directory and then restart the server")
-# Optional : Installing yarn - https://classic.yarnpkg.com/en/docs/install
-
-python setup.py compile_assets
+ --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt";
+
+## All airflow extras
+
+Airflow has a number of extras that you can install to get additional
dependencies. They sometimes install
+providers, sometimes enable other features where packages are not installed by
default.
+
+You can read more about those extras in the extras reference:
+https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html
Review Comment:
Why not pointing to local RST in
`docs/apache-airflow/extra-packages-ref.rst`?
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
-+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` |
Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+Generally none of the ``devel`` extras install providers - they expect the
providers to be used from sources
Review Comment:
Propose to clearly separate it from the overall `devel` name.
```suggestion
Generally none of the ``devel_*`` extras install providers - they expect the
providers to be used from sources
```
##########
hatch_build.py:
##########
Review Comment:
Is it good, needed or best practive to keep the "hatch_build-py" toplevel?
Would it not be better to push it down to `dev/` somehwere?
##########
docs/apache-airflow-providers/howto/create-custom-providers.rst:
##########
@@ -211,16 +211,15 @@ You need to do the following to turn an existing Python
package into a provider
.. exampleinclude:: /../../airflow/provider_info.schema.json
:language: json
-Example ``setup.cfg``:
+Example ``pyproject.toml``:
-.. code-block:: cfg
+.. code-block:: toml
- [options.entry_points]
- # the function get_provider_info is defined in myproviderpackage.somemodule
- apache_airflow_provider=
- provider_info=myproviderpackage.somemodule:get_provider_info
+ [project.entry-points."apache_airflow_provider"]
+ provider_info =
"airflow.providers.myproviderpackage.get_provider_info:get_provider_info"
Review Comment:
I believe the default Sphinx indent is three spaces (why-so-ever) - but in
our RST files I see a lot of "two spaces".
```suggestion
[project.entry-points."apache_airflow_provider"]
provider_info =
"airflow.providers.myproviderpackage.get_provider_info:get_provider_info"
```
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
-+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` |
Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+Generally none of the ``devel`` extras install providers - they expect the
providers to be used from sources
+and those extras only make sense in editable mode. Users of Airflow should not
be using them, unless they
+start contributing back and install airflow from sources.
+
+Those extras are only available in Airflow when it is installed in editable
mode from sources
+(``pip install -e .``).
+
+Devel extras
+============
+
+The devel extras do not install dependencies for features of Airflow, but add
functionality that is needed to
+develop Airflow, such as running tests, static checks. They do not install
provider packages - even if they might be related
+to some providers (like ``devel_amazon``) but they might be needed if you want
to test code of thoe corresponding
+provider.
+
+Even if some ``devel`` extras relate to providers - they do not install
provider packages - for example
+``devel_amazon`` does not install amazon provider) but they might be needed if
you want to test code of
+the corresponding provider (for example running mypy checks or running tests).
Review Comment:
paragraph has a text repetition of the paragraph before and a parenthesis
closing missing an open. Trying to re-phrase what you wanted to say? Mhm,
trying to re-phrase I assume it is just a redundant draft leftover which can be
fully deleted?
```suggestion
```
##########
pyproject.toml:
##########
@@ -14,20 +14,1472 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+[build-system]
+requires = [
+ "hatchling",
+ "gitpython",
+]
+build-backend = "hatchling.build"
+
+[project]
+name = "apache-airflow"
+dynamic = ["version"]
+
+description = " Programmatically author, schedule and monitor data pipelines"
+readme = { file = "generated/PYPI_README.md", content-type = "text/markdown" }
+license = { file = "LICENSE", content-type = "text/plain" }
+requires-python = "~=3.8"
+authors = [
+ { name = "Apache Software Foundation", email = "[email protected]" },
+]
+maintainers = [
+ {name="Apache Software Foundation", email="[email protected]"},
Review Comment:
INconsistent spacing compared to above.
```suggestion
{name = "Apache Software Foundation", email = "[email protected]"},
```
##########
hatch_build.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import logging
+import os
+from pathlib import Path
+from subprocess import run
+from typing import Any, Callable, Iterable
+
+from hatchling.builders.config import BuilderConfig
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+from hatchling.builders.plugin.interface import BuilderInterface
+from hatchling.plugin.manager import PluginManager
+
+log = logging.getLogger(__name__)
+log_level = logging.getLevelName(os.getenv("CUSTOM_AIRFLOW_BUILD_LOG_LEVEL",
"INFO"))
+log.setLevel(log_level)
+
+
+class CustomBuild(BuilderInterface[BuilderConfig, PluginManager]):
+ """Custom build class for Airflow assets and git version."""
+
+ PLUGIN_NAME = "custom"
Review Comment:
Better distinct name?
```suggestion
PLUGIN_NAME = "airflow-custom"
```
##########
setup.cfg.back:
##########
Review Comment:
I assume the file is just a local artefact and is deleted when the Draft
state of the PR is left?
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -18,17 +18,24 @@
Reference for package extras
''''''''''''''''''''''''''''
-Here's the list of all the extra dependencies of Apache Airflow.
+Airflow has a number of optional "extras" that you can use to add features to
your installation when you
+are installing Airflow. Those extras are a good way for the users to manage
their installation, but also
+they are useful for contributors to airflow when they want to contribute some
of the featuers - including
+optional integrations of Airflow - via providers.
-The entries with ``*`` in the ``Preinstalled`` column indicate that those
extras (providers) are always
-pre-installed when Airflow is installed.
+,, warning::
+
+ Traditionally in Airflow some of the extras used `.` and `_` to separate
the parts of the extra name.
+ This was not PEP-685 normalized name and we opted to change it to to `-`
for all our extras, Expecting that
+ PEP-685 will be implemented in full by `pip` and other tools. Currently
the normalization is happening
+ anyway, but `pip` shows warning when `_` or `-` are used, due to old
packaging version used (January 2023).
+ The work is in progress to change it in `this issue
<https://github.com/pypa/pip/issues/11445>` so this
+ is anticipated that it will be fixed soon.
+
+ TODO(potiuk): decide whether to do it. In the current proposal we changed
everything to `_`.
Review Comment:
Mhm, would be good to be "future ready" - do you know if this is
backwards-compatible? How about all users trying to install via `pip install
apache-airflow[apache.spark]` - if we change to `_` or `-` will this be
corrected and resolved by PIP or do we need to publish an alias as well to keep
backwards-compatability?
##########
pyproject.toml:
##########
@@ -14,20 +14,1472 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+[build-system]
+requires = [
+ "hatchling",
+ "gitpython",
+]
+build-backend = "hatchling.build"
+
+[project]
+name = "apache-airflow"
+dynamic = ["version"]
+
+description = " Programmatically author, schedule and monitor data pipelines"
Review Comment:
Is the psace as first character a "mistake" or needed?
```suggestion
description = "Programmatically author, schedule and monitor data pipelines"
```
##########
scripts/in_container/run_prepare_airflow_packages.py:
##########
@@ -79,31 +79,34 @@ def process_summary(success_message: str, error_message:
str, completed_process:
rich.print("[bright_blue]Checking airflow version\n")
airflow_version = subprocess.check_output(
- [sys.executable, "setup.py", "--version"], text=True,
cwd=AIRFLOW_SOURCES_ROOT
+ [sys.executable, "-m", "hatch", "version"], text=True,
cwd=AIRFLOW_SOURCES_ROOT
).strip()
rich.print(f"[green]Airflow version: {airflow_version}\n")
RELEASED_VERSION_MATCHER = re.compile(r"^\d+\.\d+\.\d+$")
-command = [sys.executable, "setup.py"]
+build_command = [sys.executable, "hatch", "build"]
if version_suffix:
if RELEASED_VERSION_MATCHER.match(airflow_version):
rich.print(f"[warning]Adding {version_suffix} suffix to the
{airflow_version}")
- command.extend(["egg_info", "--tag-build", version_suffix])
+ sys.exit(2)
Review Comment:
Are you sure you want to exit here? Or is this WIP code? Then the "warning"
should be converted to `[red]`
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
-+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` |
Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+Generally none of the ``devel`` extras install providers - they expect the
providers to be used from sources
+and those extras only make sense in editable mode. Users of Airflow should not
be using them, unless they
+start contributing back and install airflow from sources.
+
+Those extras are only available in Airflow when it is installed in editable
mode from sources
+(``pip install -e .``).
+
+Devel extras
+============
+
+The devel extras do not install dependencies for features of Airflow, but add
functionality that is needed to
+develop Airflow, such as running tests, static checks. They do not install
provider packages - even if they might be related
+to some providers (like ``devel_amazon``) but they might be needed if you want
to test code of thoe corresponding
+provider.
+
+Even if some ``devel`` extras relate to providers - they do not install
provider packages - for example
+``devel_amazon`` does not install amazon provider) but they might be needed if
you want to test code of
+the corresponding provider (for example running mypy checks or running tests).
+
++---------------------+-----------------------------------------+------------------------------------------------------+
+| extra | install command | enables
|
++=====================+=========================================+======================================================+
+| devel_amazon | pip install -e '.[devel_amazon]' | Adds all
test libraries needed to test amazon |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_azure | pip install -e '.[devel_azure]' | Adds all
test libraries needed to test azure |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_breeze | pip install -e '.[devel_breeze]' | Adds all
test libraries needed to test breeze |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_debuggers | pip install -e '.[devel_debuggers]' | Adds all
test libraries needed to test debuggers |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_deltalake | pip install -e '.[devel_deltalake]' | Adds all
test libraries needed to test deltalake |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_devscripts | pip install -e '.[devel_devscripts]' | Adds all
test libraries needed to test devscripts |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_duckdb | pip install -e '.[devel_duckdb]' | Adds all
test libraries needed to test duckdb |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_iceberg | pip install -e '.[devel_iceberg]' | Adds all
test libraries needed to test iceberg |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mongo | pip install -e '.[devel_mongo]' | Adds all
test libraries needed to test mongo |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mypy | pip install -e '.[devel_mypy]' | Adds all
test libraries needed to test mypy |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_sentry | pip install -e '.[devel_sentry]' | Adds all
test libraries needed to test sentry |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_static_checks | pip install -e '.[devel_static_checks]' | Adds all
test libraries needed to test static_checks |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_tests | pip install -e '.[devel_tests]' | Adds all
test libraries needed to test tests |
++---------------------+-----------------------------------------+------------------------------------------------------+
+
+Editable provider extras
+========================
+
+In order to test providers when installing Airflow in editable, development
mode, you need to install
+dependencies of the providers. This is done by installing the ``editable``
extra with ``pip install -e``.
+Those extras are not available in the released PyPI wheel packages, they are
only available when Airflow
+is installed locally in editable mode.
+
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| extra | install command
| enables |
++=================================+=====================================================+============================================================+
+| editable_airbyte | pip install -e '.[editable_airbyte]'
| Adds all libraries needed by the airbyte provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_alibaba | pip install -e '.[editable_alibaba]'
| Adds all libraries needed by the alibaba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_amazon | pip install -e '.[editable_amazon]'
| Adds all libraries needed by the amazon provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_beam | pip install -e '.[editable_apache_beam]'
| Adds all libraries needed by the apache_beam provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_cassandra | pip install -e
'.[editable_apache_cassandra]' | Adds all libraries needed by the
apache_cassandra provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_drill | pip install -e '.[editable_apache_drill]'
| Adds all libraries needed by the apache_drill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_druid | pip install -e '.[editable_apache_druid]'
| Adds all libraries needed by the apache_druid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_flink | pip install -e '.[editable_apache_flink]'
| Adds all libraries needed by the apache_flink provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hdfs | pip install -e '.[editable_apache_hdfs]'
| Adds all libraries needed by the apache_hdfs provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hive | pip install -e '.[editable_apache_hive]'
| Adds all libraries needed by the apache_hive provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_impala | pip install -e '.[editable_apache_impala]'
| Adds all libraries needed by the apache_impala provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kafka | pip install -e '.[editable_apache_kafka]'
| Adds all libraries needed by the apache_kafka provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kylin | pip install -e '.[editable_apache_kylin]'
| Adds all libraries needed by the apache_kylin provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_livy | pip install -e '.[editable_apache_livy]'
| Adds all libraries needed by the apache_livy provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pig | pip install -e '.[editable_apache_pig]'
| Adds all libraries needed by the apache_pig provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pinot | pip install -e '.[editable_apache_pinot]'
| Adds all libraries needed by the apache_pinot provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_spark | pip install -e '.[editable_apache_spark]'
| Adds all libraries needed by the apache_spark provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apprise | pip install -e '.[editable_apprise]'
| Adds all libraries needed by the apprise provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_arangodb | pip install -e '.[editable_arangodb]'
| Adds all libraries needed by the arangodb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_asana | pip install -e '.[editable_asana]'
| Adds all libraries needed by the asana provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_atlassian_jira | pip install -e
'.[editable_atlassian_jira]' | Adds all libraries needed by the
atlassian_jira provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_celery | pip install -e '.[editable_celery]'
| Adds all libraries needed by the celery provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cloudant | pip install -e '.[editable_cloudant]'
| Adds all libraries needed by the cloudant provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cncf_kubernetes | pip install -e
'.[editable_cncf_kubernetes]' | Adds all libraries needed by the
cncf_kubernetes provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cohere | pip install -e '.[editable_cohere]'
| Adds all libraries needed by the cohere provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_io | pip install -e '.[editable_common_io]'
| Adds all libraries needed by the common_io provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_sql | pip install -e '.[editable_common_sql]'
| Adds all libraries needed by the common_sql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_databricks | pip install -e '.[editable_databricks]'
| Adds all libraries needed by the databricks provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_datadog | pip install -e '.[editable_datadog]'
| Adds all libraries needed by the datadog provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dbt_cloud | pip install -e '.[editable_dbt_cloud]'
| Adds all libraries needed by the dbt_cloud provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dingding | pip install -e '.[editable_dingding]'
| Adds all libraries needed by the dingding provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_discord | pip install -e '.[editable_discord]'
| Adds all libraries needed by the discord provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_docker | pip install -e '.[editable_docker]'
| Adds all libraries needed by the docker provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_elasticsearch | pip install -e '.[editable_elasticsearch]'
| Adds all libraries needed by the elasticsearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_exasol | pip install -e '.[editable_exasol]'
| Adds all libraries needed by the exasol provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_fab | pip install -e '.[editable_fab]'
| Adds all libraries needed by the fab provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_facebook | pip install -e '.[editable_facebook]'
| Adds all libraries needed by the facebook provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ftp | pip install -e '.[editable_ftp]'
| Adds all libraries needed by the ftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_github | pip install -e '.[editable_github]'
| Adds all libraries needed by the github provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_google | pip install -e '.[editable_google]'
| Adds all libraries needed by the google provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_grpc | pip install -e '.[editable_grpc]'
| Adds all libraries needed by the grpc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_hashicorp | pip install -e '.[editable_hashicorp]'
| Adds all libraries needed by the hashicorp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_http | pip install -e '.[editable_http]'
| Adds all libraries needed by the http provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_imap | pip install -e '.[editable_imap]'
| Adds all libraries needed by the imap provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_influxdb | pip install -e '.[editable_influxdb]'
| Adds all libraries needed by the influxdb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jdbc | pip install -e '.[editable_jdbc]'
| Adds all libraries needed by the jdbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jenkins | pip install -e '.[editable_jenkins]'
| Adds all libraries needed by the jenkins provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_azure | pip install -e
'.[editable_microsoft_azure]' | Adds all libraries needed by the
microsoft_azure provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_mssql | pip install -e
'.[editable_microsoft_mssql]' | Adds all libraries needed by the
microsoft_mssql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_psrp | pip install -e
'.[editable_microsoft_psrp]' | Adds all libraries needed by the
microsoft_psrp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_winrm | pip install -e
'.[editable_microsoft_winrm]' | Adds all libraries needed by the
microsoft_winrm provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mongo | pip install -e '.[editable_mongo]'
| Adds all libraries needed by the mongo provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mysql | pip install -e '.[editable_mysql]'
| Adds all libraries needed by the mysql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_neo4j | pip install -e '.[editable_neo4j]'
| Adds all libraries needed by the neo4j provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_odbc | pip install -e '.[editable_odbc]'
| Adds all libraries needed by the odbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openai | pip install -e '.[editable_openai]'
| Adds all libraries needed by the openai provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openfaas | pip install -e '.[editable_openfaas]'
| Adds all libraries needed by the openfaas provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openlineage | pip install -e '.[editable_openlineage]'
| Adds all libraries needed by the openlineage provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opensearch | pip install -e '.[editable_opensearch]'
| Adds all libraries needed by the opensearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opsgenie | pip install -e '.[editable_opsgenie]'
| Adds all libraries needed by the opsgenie provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_oracle | pip install -e '.[editable_oracle]'
| Adds all libraries needed by the oracle provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pagerduty | pip install -e '.[editable_pagerduty]'
| Adds all libraries needed by the pagerduty provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_papermill | pip install -e '.[editable_papermill]'
| Adds all libraries needed by the papermill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pgvector | pip install -e '.[editable_pgvector]'
| Adds all libraries needed by the pgvector provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pinecone | pip install -e '.[editable_pinecone]'
| Adds all libraries needed by the pinecone provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_postgres | pip install -e '.[editable_postgres]'
| Adds all libraries needed by the postgres provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_presto | pip install -e '.[editable_presto]'
| Adds all libraries needed by the presto provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_redis | pip install -e '.[editable_redis]'
| Adds all libraries needed by the redis provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_salesforce | pip install -e '.[editable_salesforce]'
| Adds all libraries needed by the salesforce provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_samba | pip install -e '.[editable_samba]'
| Adds all libraries needed by the samba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_segment | pip install -e '.[editable_segment]'
| Adds all libraries needed by the segment provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sendgrid | pip install -e '.[editable_sendgrid]'
| Adds all libraries needed by the sendgrid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sftp | pip install -e '.[editable_sftp]'
| Adds all libraries needed by the sftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_singularity | pip install -e '.[editable_singularity]'
| Adds all libraries needed by the singularity provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_slack | pip install -e '.[editable_slack]'
| Adds all libraries needed by the slack provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_smtp | pip install -e '.[editable_smtp]'
| Adds all libraries needed by the smtp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_snowflake | pip install -e '.[editable_snowflake]'
| Adds all libraries needed by the snowflake provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sqlite | pip install -e '.[editable_sqlite]'
| Adds all libraries needed by the sqlite provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ssh | pip install -e '.[editable_ssh]'
| Adds all libraries needed by the ssh provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tableau | pip install -e '.[editable_tableau]'
| Adds all libraries needed by the tableau provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tabular | pip install -e '.[editable_tabular]'
| Adds all libraries needed by the tabular provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_telegram | pip install -e '.[editable_telegram]'
| Adds all libraries needed by the telegram provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_trino | pip install -e '.[editable_trino]'
| Adds all libraries needed by the trino provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_vertica | pip install -e '.[editable_vertica]'
| Adds all libraries needed by the vertica provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_weaviate | pip install -e '.[editable_weaviate]'
| Adds all libraries needed by the weaviate provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_yandex | pip install -e '.[editable_yandex]'
| Adds all libraries needed by the yandex provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_zendesk | pip install -e '.[editable_zendesk]'
| Adds all libraries needed by the zendesk provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+
+Doc extras
+==========
+
+Those are the extras that are needed to generated documentation for Airflow.
This is used for development time only
+
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_all | ``pip install 'apache-airflow[devel_all]'`` |
Everything needed for development including Hadoop and providers |
+| extra | install command |
enables |
++=====================+=====================================================+========================================================================+
+| doc | ``pip install -e '.[doc]'`` |
Packages needed to build docs (included in ``devel``) |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_ci | ``pip install 'apache-airflow[devel_ci]'`` |
All dependencies required for CI tests (same as ``devel_all``) |
+| doc_gen | ``pip install -e '.[doc_gen]'`` |
Packages needed to generate er diagrams (included in ``devel_all``) |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-Doc extras
-----------
+Bundle devel extras
+===================
-Those are the extras that are needed to generated documentation for Airflow.
This is used for development time only
+Those are extras that bundle devel, editable and doc extras together to make
it easy to install them together in a single installation. Some of the
+extras are more difficult to install on certain systems (such as ARM MacBooks)
because they require system level dependencies to be installed.
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| extra | install command |
enables |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| doc | ``pip install 'apache-airflow[doc]'`` |
Packages needed to build docs (included in ``devel``) |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| doc_gen | ``pip install 'apache-airflow[doc_gen]'`` |
Packages needed to generate er diagrams (included in ``devel_all``) |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| extra | install command |
enables |
++=====================+=====================================================+========================================================================+
+| devel | ``pip install -e '.[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| devel_hadoop | ``pip install -e '.[devel_hadoop]'`` |
Adds Hadoop stack libraries ``devel`` dependencies |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| devel_all_dbs | ``pip install -e '.[devel_all_dbs]'`` |
Adds all libraries (editable extras) needed to test database providers |
Review Comment:
+1 here for "all databases :-D
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
-+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` |
Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+Generally none of the ``devel`` extras install providers - they expect the
providers to be used from sources
+and those extras only make sense in editable mode. Users of Airflow should not
be using them, unless they
+start contributing back and install airflow from sources.
+
+Those extras are only available in Airflow when it is installed in editable
mode from sources
+(``pip install -e .``).
+
+Devel extras
+============
+
+The devel extras do not install dependencies for features of Airflow, but add
functionality that is needed to
+develop Airflow, such as running tests, static checks. They do not install
provider packages - even if they might be related
+to some providers (like ``devel_amazon``) but they might be needed if you want
to test code of thoe corresponding
+provider.
+
+Even if some ``devel`` extras relate to providers - they do not install
provider packages - for example
+``devel_amazon`` does not install amazon provider) but they might be needed if
you want to test code of
+the corresponding provider (for example running mypy checks or running tests).
+
++---------------------+-----------------------------------------+------------------------------------------------------+
+| extra | install command | enables
|
++=====================+=========================================+======================================================+
+| devel_amazon | pip install -e '.[devel_amazon]' | Adds all
test libraries needed to test amazon |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_azure | pip install -e '.[devel_azure]' | Adds all
test libraries needed to test azure |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_breeze | pip install -e '.[devel_breeze]' | Adds all
test libraries needed to test breeze |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_debuggers | pip install -e '.[devel_debuggers]' | Adds all
test libraries needed to test debuggers |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_deltalake | pip install -e '.[devel_deltalake]' | Adds all
test libraries needed to test deltalake |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_devscripts | pip install -e '.[devel_devscripts]' | Adds all
test libraries needed to test devscripts |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_duckdb | pip install -e '.[devel_duckdb]' | Adds all
test libraries needed to test duckdb |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_iceberg | pip install -e '.[devel_iceberg]' | Adds all
test libraries needed to test iceberg |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mongo | pip install -e '.[devel_mongo]' | Adds all
test libraries needed to test mongo |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mypy | pip install -e '.[devel_mypy]' | Adds all
test libraries needed to test mypy |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_sentry | pip install -e '.[devel_sentry]' | Adds all
test libraries needed to test sentry |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_static_checks | pip install -e '.[devel_static_checks]' | Adds all
test libraries needed to test static_checks |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_tests | pip install -e '.[devel_tests]' | Adds all
test libraries needed to test tests |
++---------------------+-----------------------------------------+------------------------------------------------------+
+
+Editable provider extras
+========================
+
+In order to test providers when installing Airflow in editable, development
mode, you need to install
+dependencies of the providers. This is done by installing the ``editable``
extra with ``pip install -e``.
+Those extras are not available in the released PyPI wheel packages, they are
only available when Airflow
+is installed locally in editable mode.
+
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| extra | install command
| enables |
++=================================+=====================================================+============================================================+
+| editable_airbyte | pip install -e '.[editable_airbyte]'
| Adds all libraries needed by the airbyte provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_alibaba | pip install -e '.[editable_alibaba]'
| Adds all libraries needed by the alibaba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_amazon | pip install -e '.[editable_amazon]'
| Adds all libraries needed by the amazon provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_beam | pip install -e '.[editable_apache_beam]'
| Adds all libraries needed by the apache_beam provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_cassandra | pip install -e
'.[editable_apache_cassandra]' | Adds all libraries needed by the
apache_cassandra provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_drill | pip install -e '.[editable_apache_drill]'
| Adds all libraries needed by the apache_drill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_druid | pip install -e '.[editable_apache_druid]'
| Adds all libraries needed by the apache_druid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_flink | pip install -e '.[editable_apache_flink]'
| Adds all libraries needed by the apache_flink provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hdfs | pip install -e '.[editable_apache_hdfs]'
| Adds all libraries needed by the apache_hdfs provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hive | pip install -e '.[editable_apache_hive]'
| Adds all libraries needed by the apache_hive provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_impala | pip install -e '.[editable_apache_impala]'
| Adds all libraries needed by the apache_impala provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kafka | pip install -e '.[editable_apache_kafka]'
| Adds all libraries needed by the apache_kafka provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kylin | pip install -e '.[editable_apache_kylin]'
| Adds all libraries needed by the apache_kylin provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_livy | pip install -e '.[editable_apache_livy]'
| Adds all libraries needed by the apache_livy provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pig | pip install -e '.[editable_apache_pig]'
| Adds all libraries needed by the apache_pig provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pinot | pip install -e '.[editable_apache_pinot]'
| Adds all libraries needed by the apache_pinot provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_spark | pip install -e '.[editable_apache_spark]'
| Adds all libraries needed by the apache_spark provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apprise | pip install -e '.[editable_apprise]'
| Adds all libraries needed by the apprise provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_arangodb | pip install -e '.[editable_arangodb]'
| Adds all libraries needed by the arangodb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_asana | pip install -e '.[editable_asana]'
| Adds all libraries needed by the asana provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_atlassian_jira | pip install -e
'.[editable_atlassian_jira]' | Adds all libraries needed by the
atlassian_jira provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_celery | pip install -e '.[editable_celery]'
| Adds all libraries needed by the celery provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cloudant | pip install -e '.[editable_cloudant]'
| Adds all libraries needed by the cloudant provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cncf_kubernetes | pip install -e
'.[editable_cncf_kubernetes]' | Adds all libraries needed by the
cncf_kubernetes provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cohere | pip install -e '.[editable_cohere]'
| Adds all libraries needed by the cohere provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_io | pip install -e '.[editable_common_io]'
| Adds all libraries needed by the common_io provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_sql | pip install -e '.[editable_common_sql]'
| Adds all libraries needed by the common_sql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_databricks | pip install -e '.[editable_databricks]'
| Adds all libraries needed by the databricks provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_datadog | pip install -e '.[editable_datadog]'
| Adds all libraries needed by the datadog provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dbt_cloud | pip install -e '.[editable_dbt_cloud]'
| Adds all libraries needed by the dbt_cloud provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dingding | pip install -e '.[editable_dingding]'
| Adds all libraries needed by the dingding provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_discord | pip install -e '.[editable_discord]'
| Adds all libraries needed by the discord provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_docker | pip install -e '.[editable_docker]'
| Adds all libraries needed by the docker provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_elasticsearch | pip install -e '.[editable_elasticsearch]'
| Adds all libraries needed by the elasticsearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_exasol | pip install -e '.[editable_exasol]'
| Adds all libraries needed by the exasol provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_fab | pip install -e '.[editable_fab]'
| Adds all libraries needed by the fab provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_facebook | pip install -e '.[editable_facebook]'
| Adds all libraries needed by the facebook provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ftp | pip install -e '.[editable_ftp]'
| Adds all libraries needed by the ftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_github | pip install -e '.[editable_github]'
| Adds all libraries needed by the github provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_google | pip install -e '.[editable_google]'
| Adds all libraries needed by the google provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_grpc | pip install -e '.[editable_grpc]'
| Adds all libraries needed by the grpc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_hashicorp | pip install -e '.[editable_hashicorp]'
| Adds all libraries needed by the hashicorp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_http | pip install -e '.[editable_http]'
| Adds all libraries needed by the http provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_imap | pip install -e '.[editable_imap]'
| Adds all libraries needed by the imap provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_influxdb | pip install -e '.[editable_influxdb]'
| Adds all libraries needed by the influxdb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jdbc | pip install -e '.[editable_jdbc]'
| Adds all libraries needed by the jdbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jenkins | pip install -e '.[editable_jenkins]'
| Adds all libraries needed by the jenkins provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_azure | pip install -e
'.[editable_microsoft_azure]' | Adds all libraries needed by the
microsoft_azure provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_mssql | pip install -e
'.[editable_microsoft_mssql]' | Adds all libraries needed by the
microsoft_mssql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_psrp | pip install -e
'.[editable_microsoft_psrp]' | Adds all libraries needed by the
microsoft_psrp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_winrm | pip install -e
'.[editable_microsoft_winrm]' | Adds all libraries needed by the
microsoft_winrm provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mongo | pip install -e '.[editable_mongo]'
| Adds all libraries needed by the mongo provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mysql | pip install -e '.[editable_mysql]'
| Adds all libraries needed by the mysql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_neo4j | pip install -e '.[editable_neo4j]'
| Adds all libraries needed by the neo4j provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_odbc | pip install -e '.[editable_odbc]'
| Adds all libraries needed by the odbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openai | pip install -e '.[editable_openai]'
| Adds all libraries needed by the openai provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openfaas | pip install -e '.[editable_openfaas]'
| Adds all libraries needed by the openfaas provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openlineage | pip install -e '.[editable_openlineage]'
| Adds all libraries needed by the openlineage provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opensearch | pip install -e '.[editable_opensearch]'
| Adds all libraries needed by the opensearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opsgenie | pip install -e '.[editable_opsgenie]'
| Adds all libraries needed by the opsgenie provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_oracle | pip install -e '.[editable_oracle]'
| Adds all libraries needed by the oracle provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pagerduty | pip install -e '.[editable_pagerduty]'
| Adds all libraries needed by the pagerduty provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_papermill | pip install -e '.[editable_papermill]'
| Adds all libraries needed by the papermill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pgvector | pip install -e '.[editable_pgvector]'
| Adds all libraries needed by the pgvector provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pinecone | pip install -e '.[editable_pinecone]'
| Adds all libraries needed by the pinecone provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_postgres | pip install -e '.[editable_postgres]'
| Adds all libraries needed by the postgres provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_presto | pip install -e '.[editable_presto]'
| Adds all libraries needed by the presto provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_redis | pip install -e '.[editable_redis]'
| Adds all libraries needed by the redis provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_salesforce | pip install -e '.[editable_salesforce]'
| Adds all libraries needed by the salesforce provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_samba | pip install -e '.[editable_samba]'
| Adds all libraries needed by the samba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_segment | pip install -e '.[editable_segment]'
| Adds all libraries needed by the segment provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sendgrid | pip install -e '.[editable_sendgrid]'
| Adds all libraries needed by the sendgrid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sftp | pip install -e '.[editable_sftp]'
| Adds all libraries needed by the sftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_singularity | pip install -e '.[editable_singularity]'
| Adds all libraries needed by the singularity provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_slack | pip install -e '.[editable_slack]'
| Adds all libraries needed by the slack provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_smtp | pip install -e '.[editable_smtp]'
| Adds all libraries needed by the smtp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_snowflake | pip install -e '.[editable_snowflake]'
| Adds all libraries needed by the snowflake provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sqlite | pip install -e '.[editable_sqlite]'
| Adds all libraries needed by the sqlite provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ssh | pip install -e '.[editable_ssh]'
| Adds all libraries needed by the ssh provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tableau | pip install -e '.[editable_tableau]'
| Adds all libraries needed by the tableau provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tabular | pip install -e '.[editable_tabular]'
| Adds all libraries needed by the tabular provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_telegram | pip install -e '.[editable_telegram]'
| Adds all libraries needed by the telegram provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_trino | pip install -e '.[editable_trino]'
| Adds all libraries needed by the trino provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_vertica | pip install -e '.[editable_vertica]'
| Adds all libraries needed by the vertica provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_weaviate | pip install -e '.[editable_weaviate]'
| Adds all libraries needed by the weaviate provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_yandex | pip install -e '.[editable_yandex]'
| Adds all libraries needed by the yandex provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_zendesk | pip install -e '.[editable_zendesk]'
| Adds all libraries needed by the zendesk provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+
+Doc extras
Review Comment:
As `devel` and `devel_all` is described in the chapter below I'd propose to
switch order such that the "Doc extras" are below "Bundle devel extras" or
merge both lists as they are actually belonging close together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
