pankajkoti commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1229356103
########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -40,33 +40,65 @@ Only ``pip`` installation is currently officially supported. the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that newer versions of ``bazel`` will handle it. -Typical command to install airflow from PyPI looks like below: +Typical command to install airflow from scratch in a reproducible way from PyPI looks like below: -.. code-block:: +.. code-block:: bash pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" -This is an example, see further for more explanation. + +Typically, you can add other dependencies and providers as separate command after the reproducible +installation - this way you can upgrade or downgrade the dependencies as you see fit, without limiting them to +constraints. Good practice for those is to extend such ``pip install`` command with the ``apache-airflow`` +pinned to the version you have already installed to make sure it is not accidentally +upgraded or downgraded by ``pip``. + +.. code-block:: bash + + pip install "apache-airflow==|version|" apache-airflow-providers-google=10.1.0 Review Comment: ```suggestion pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.0 ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -40,33 +40,65 @@ Only ``pip`` installation is currently officially supported. the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that newer versions of ``bazel`` will handle it. -Typical command to install airflow from PyPI looks like below: +Typical command to install airflow from scratch in a reproducible way from PyPI looks like below: -.. code-block:: +.. code-block:: bash pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" -This is an example, see further for more explanation. + +Typically, you can add other dependencies and providers as separate command after the reproducible +installation - this way you can upgrade or downgrade the dependencies as you see fit, without limiting them to +constraints. Good practice for those is to extend such ``pip install`` command with the ``apache-airflow`` +pinned to the version you have already installed to make sure it is not accidentally +upgraded or downgraded by ``pip``. + +.. code-block:: bash + + pip install "apache-airflow==|version|" apache-airflow-providers-google=10.1.0 + + +Those are just examples, see further for more explanation why those are the best practices. + +.. note:: + + Generally speaking, Python community established practice is to perform application installation in a + virtualenv created with ``virtualenv`` or ``venv`` tools. You can also use ``pipx`` to install Airflow in a + application dedicated virtual environment created for you. There are also other tools that can be used + to manage your virtualenv installation and you are free to choose how you are managing the environments, Review Comment: ```suggestion to manage your virtualenv installation and you are free to choose how you are managing the environments. ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client + libraries in case you install a database provider, etc.. You need to figure out which system dependencies + you need when your installation fails and install them before retrying the installation. + +Upgrading and installing dependencies (including providers) +=========================================================== + +**The reproducible installation above should not prevent you from being able to upgrade or downgrade +providers and other dependencies to other versions** - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt +You can, for example, install new versions of providers and dependencies after the release to use the latest +version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version. +Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility +reasons. Installing such dependencies should be done without constraints as a separate pip command. +When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of +packages to install and pin it to the version that you have, otherwise you might end up with a +different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when Review Comment: ```suggestion different version of Airflow than you expect because ``pip`` can upgrade/downgrade it automatically when ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client + libraries in case you install a database provider, etc.. You need to figure out which system dependencies + you need when your installation fails and install them before retrying the installation. + +Upgrading and installing dependencies (including providers) +=========================================================== + +**The reproducible installation above should not prevent you from being able to upgrade or downgrade +providers and other dependencies to other versions** - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt +You can, for example, install new versions of providers and dependencies after the release to use the latest +version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version. +Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility +reasons. Installing such dependencies should be done without constraints as a separate pip command. +When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of +packages to install and pin it to the version that you have, otherwise you might end up with a +different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when +performing dependency resolution. + + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1 -Fixing Constraint files at release time -''''''''''''''''''''''''''''''''''''''' +You can also downgrade or upgrade other dependencies this way - even if they are not compatible with +those dependencies that are stored in the original constraints file: + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + +.. warning:: + + Not all dependencies can be installed this way - you might have dependencies conflicting with basic + requirements of Airflow or other dependencies installed in your system. However, by skipping constraints + when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you, + while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require. + The resulting combination of those dependencies might not be tested together before as well as the set + of dependencies that come with constraints, but it should work in most cases as we usually add + requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot + install some dependencies in the same environment as Airflow - you can attempt to use other approaches. + See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>` + + +Verifying installed dependencies +================================ + +You can also always run the ``pip check`` command to test if the set of your Python packages is +consistent and not conflicting. + + +.. code-block:: bash + + > pip check + No broken requirements found. + + +When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no +conflicting dependencies in your environment. + + +Using your own constraints +========================== + +When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want +to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it +to install Airflow instead of the one provided by the community. + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" dbt-core==0.20.0 + pip freeze > my-constraints.txt Review Comment: I like this section a lot 👍🏽 ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -197,11 +296,25 @@ that conflicts with the version of apache-airflow that you are using: pip install "apache-airflow==|version|" "apache-airflow-providers-google==8.0.0" +.. note:: + + Installing, upgrading, downgrading providers separately is not guaranteed to work with all + Airflow versions or other providers. Some providers have minimum-required version of Airflow and some + versions of providers might have conflicting requirements with Airflow or other dependencies you Review Comment: could we please improvise this statement "Some versions of providers might have conflicting requirements with Airflow" a bit? Do we mean some versions of Airflow here or we meant conflicting requirements with other Airflow providers? Otherwise, this might be questioned if it is an Airflow provider why or in which cases it would have conflicting requirements with Airflow? ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client + libraries in case you install a database provider, etc.. You need to figure out which system dependencies + you need when your installation fails and install them before retrying the installation. + +Upgrading and installing dependencies (including providers) +=========================================================== + +**The reproducible installation above should not prevent you from being able to upgrade or downgrade +providers and other dependencies to other versions** - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt +You can, for example, install new versions of providers and dependencies after the release to use the latest +version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version. +Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility +reasons. Installing such dependencies should be done without constraints as a separate pip command. +When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of +packages to install and pin it to the version that you have, otherwise you might end up with a +different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when +performing dependency resolution. + + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1 -Fixing Constraint files at release time -''''''''''''''''''''''''''''''''''''''' +You can also downgrade or upgrade other dependencies this way - even if they are not compatible with +those dependencies that are stored in the original constraints file: + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + +.. warning:: + + Not all dependencies can be installed this way - you might have dependencies conflicting with basic + requirements of Airflow or other dependencies installed in your system. However, by skipping constraints + when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you, + while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require. + The resulting combination of those dependencies might not be tested together before as well as the set + of dependencies that come with constraints, but it should work in most cases as we usually add Review Comment: ```suggestion The resulting combination of those dependencies and the set of dependencies that come with the constraints might not be tested before, but it should work in most cases as we usually add ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client Review Comment: ```suggestion be installed such as ``build-essential`` in order to compile libraries, or for example database client ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client + libraries in case you install a database provider, etc.. You need to figure out which system dependencies + you need when your installation fails and install them before retrying the installation. + +Upgrading and installing dependencies (including providers) +=========================================================== + +**The reproducible installation above should not prevent you from being able to upgrade or downgrade +providers and other dependencies to other versions** - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt +You can, for example, install new versions of providers and dependencies after the release to use the latest +version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version. +Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility +reasons. Installing such dependencies should be done without constraints as a separate pip command. +When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of +packages to install and pin it to the version that you have, otherwise you might end up with a +different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when +performing dependency resolution. + + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1 -Fixing Constraint files at release time -''''''''''''''''''''''''''''''''''''''' +You can also downgrade or upgrade other dependencies this way - even if they are not compatible with +those dependencies that are stored in the original constraints file: + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + +.. warning:: + + Not all dependencies can be installed this way - you might have dependencies conflicting with basic + requirements of Airflow or other dependencies installed in your system. However, by skipping constraints + when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you, + while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require. + The resulting combination of those dependencies might not be tested together before as well as the set + of dependencies that come with constraints, but it should work in most cases as we usually add + requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot + install some dependencies in the same environment as Airflow - you can attempt to use other approaches. + See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>` + + +Verifying installed dependencies +================================ + +You can also always run the ``pip check`` command to test if the set of your Python packages is +consistent and not conflicting. + + +.. code-block:: bash + + > pip check + No broken requirements found. + + +When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no +conflicting dependencies in your environment. + + +Using your own constraints +========================== + +When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want +to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it +to install Airflow instead of the one provided by the community. Review Comment: ```suggestion to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it to install Airflow instead of the one provided by the community. ``` Just suggesting a new line char as I see this line is relatively longer than all other lines :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
