o-nikolas commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228802464
########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -53,20 +53,28 @@ This is an example, see further for more explanation. Constraints files ''''''''''''''''' -Airflow installation can be tricky sometimes because Airflow is both a library and an application. +Why do we need constraints Review Comment: The below suggestion is more of a statement so it doesn't need a `?` at the end. ```suggestion Why we need constraints ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +However, that should not prevent you from being able to install providers and dependencies that were upgraded +after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version. +Installing such dependencies should be done without constraints as a separate pip command. -You can create the URL to the file substituting the variables in the template below. +You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin +it to the version that you have, otherwise you might end up with a different version of Airflow than you +expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution. -.. code-block:: +.. code-block:: bash - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1 -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +You can also downgrade providers or other dependencies this way if you need to (for example if you do not +want to upgrade to the latest version of a provider). -.. code-block:: +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + + +Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints, +you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies +that Apache Airflow and other providers require. The combination of those dependencies might not be as well +tested as the "golden" set of dependencies, but it should work in most cases. You can also always run +the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting. + + +Using your own constraints Review Comment: Love this section, it should help lots of folks :+1: ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -53,20 +53,28 @@ This is an example, see further for more explanation. Constraints files ''''''''''''''''' -Airflow installation can be tricky sometimes because Airflow is both a library and an application. +Why do we need constraints +========================== + +Airflow installation can be tricky because Airflow is both a library and an application. + Libraries usually keep their dependencies open and applications usually pin them, but we should do neither and both at the same time. We decided to keep our dependencies as open as possible (in ``setup.cfg`` and ``setup.py``) so users can install different version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will not work or will produce an unusable Airflow installation. -In order to have a repeatable installation (and only for that reason), we also keep a set of "known-to-be-working" constraint files in the +Reproducible Airflow installation +================================= + +In order to have a reproducible installation, we also keep a set of constraint files in the ``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag -for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies. +for each released version e.g. :subst-code:`constraints-|version|`. -Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint -files when installing Airflow from PyPI. Note that you have to specify the correct Airflow -and Python versions in the URL. +This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability +of having the exact same installation of airflow + providers + dependencies as was known to be working +at the moment of release - "golden" set of dependencies for that version of Airflow. There is a separate Review Comment: I'd drop the golden piece. It makes it sound as overly desirable and useful in all cases, which may lead to some of the confusion we're trying to avoid. ```suggestion This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability of having the exact same installation of airflow + providers + dependencies as was known to be working at the moment of release. There is a separate ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +However, that should not prevent you from being able to install providers and dependencies that were upgraded +after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version. +Installing such dependencies should be done without constraints as a separate pip command. Review Comment: What do you think about bolding this line to really make it stand out and hit the point home? ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +However, that should not prevent you from being able to install providers and dependencies that were upgraded +after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version. +Installing such dependencies should be done without constraints as a separate pip command. -You can create the URL to the file substituting the variables in the template below. +You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin +it to the version that you have, otherwise you might end up with a different version of Airflow than you +expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution. -.. code-block:: +.. code-block:: bash - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1 -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +You can also downgrade providers or other dependencies this way if you need to (for example if you do not +want to upgrade to the latest version of a provider). -.. code-block:: +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + + +Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints, +you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies +that Apache Airflow and other providers require. The combination of those dependencies might not be as well +tested as the "golden" set of dependencies, but it should work in most cases. You can also always run +the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting. + + +Using your own constraints +========================== + +When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want +to continue being able to run reproducible installation of Airflow and those dependencies. In order to do +that you can produce your own constraints file and use it to install Airflow instead of the one provided +by the community. + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + pip freeze > my-constraints.txt + + +Then you can use it to run reproducible installation of your environment in a single operation via local file: Review Comment: ```suggestion Then you can use it to create reproducible installations of your environment in a single operation via a local constraints file: ``` ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +However, that should not prevent you from being able to install providers and dependencies that were upgraded +after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version. +Installing such dependencies should be done without constraints as a separate pip command. -You can create the URL to the file substituting the variables in the template below. +You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin +it to the version that you have, otherwise you might end up with a different version of Airflow than you +expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution. -.. code-block:: +.. code-block:: bash - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1 -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +You can also downgrade providers or other dependencies this way if you need to (for example if you do not +want to upgrade to the latest version of a provider). -.. code-block:: +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + + +Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints, +you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies +that Apache Airflow and other providers require. The combination of those dependencies might not be as well +tested as the "golden" set of dependencies, but it should work in most cases. You can also always run +the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting. + + +Using your own constraints +========================== + +When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want +to continue being able to run reproducible installation of Airflow and those dependencies. In order to do +that you can produce your own constraints file and use it to install Airflow instead of the one provided +by the community. + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + pip freeze > my-constraints.txt + + +Then you can use it to run reproducible installation of your environment in a single operation via local file: + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt" + + +The constraint file might also be hosted via a webserver of your choice and made available for remote use: + + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://my-company.org/my-constraints.txt"; - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt - - -Fixing Constraint files at release time Review Comment: I think this header and the first paragraph are worth keeping to set expectations about the constraints files. ########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +However, that should not prevent you from being able to install providers and dependencies that were upgraded +after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version. +Installing such dependencies should be done without constraints as a separate pip command. -You can create the URL to the file substituting the variables in the template below. +You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin +it to the version that you have, otherwise you might end up with a different version of Airflow than you +expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution. -.. code-block:: +.. code-block:: bash - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1 -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +You can also downgrade providers or other dependencies this way if you need to (for example if you do not +want to upgrade to the latest version of a provider). -.. code-block:: +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + + +Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints, Review Comment: Such an upgrade is not always possible - you might have conflicting dependencies, but by not using constraints, ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
