potiuk commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1229636516
########## docs/apache-airflow/installation/installing-from-pypi.rst: ########## @@ -79,50 +111,124 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9`` -There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to -install Airflow core. This allows to install and upgrade airflow separately and independently from providers. +The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra, +but you can pick your own set of extras and providers to install. -You can create the URL to the file substituting the variables in the template below. +.. code-block:: bash -.. code-block:: + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" - https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt -You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" -constraints always points to the "latest" released Airflow version constraints: +.. note:: -.. code-block:: + The reproducible installation guarantees that this initial installation steps will always work for you - + providing that you use the right Python version and that you have appropriate Operating System dependencies + installed for the providers to be installed. Some of the providers require additional OS dependencies to + be installed such us ``build-essential`` in order to compile libraries, or for example database client + libraries in case you install a database provider, etc.. You need to figure out which system dependencies + you need when your installation fails and install them before retrying the installation. + +Upgrading and installing dependencies (including providers) +=========================================================== + +**The reproducible installation above should not prevent you from being able to upgrade or downgrade +providers and other dependencies to other versions** - https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt +You can, for example, install new versions of providers and dependencies after the release to use the latest +version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version. +Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility +reasons. Installing such dependencies should be done without constraints as a separate pip command. +When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of +packages to install and pin it to the version that you have, otherwise you might end up with a +different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when +performing dependency resolution. + + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1 -Fixing Constraint files at release time -''''''''''''''''''''''''''''''''''''''' +You can also downgrade or upgrade other dependencies this way - even if they are not compatible with +those dependencies that are stored in the original constraints file: + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 + +.. warning:: + + Not all dependencies can be installed this way - you might have dependencies conflicting with basic + requirements of Airflow or other dependencies installed in your system. However, by skipping constraints + when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you, + while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require. + The resulting combination of those dependencies might not be tested together before as well as the set + of dependencies that come with constraints, but it should work in most cases as we usually add + requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot + install some dependencies in the same environment as Airflow - you can attempt to use other approaches. + See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>` + + +Verifying installed dependencies +================================ + +You can also always run the ``pip check`` command to test if the set of your Python packages is +consistent and not conflicting. + + +.. code-block:: bash + + > pip check + No broken requirements found. + + +When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no +conflicting dependencies in your environment. + + +Using your own constraints +========================== + +When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want +to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it +to install Airflow instead of the one provided by the community. + +.. code-block:: bash + + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt" + pip install "apache-airflow==|version|" dbt-core==0.20.0 + pip freeze > my-constraints.txt Review Comment: This is what a few people at the London meetup told as well. For me that was pretty obvious (It's how constraints were generated from the very beginning) but with the recent discussion and the talk I gave I realized that It was only "my bubble". People were - apparently so far- manually modifying the constraints and complained for example that recent google provider is not compatible with their (it turned out) custom constraints. Funny how some things obvious to you might be completely alien to others :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
