o-nikolas commented on code in PR #37796:
URL: https://github.com/apache/airflow/pull/37796#discussion_r1511899587
##########
contributing-docs/testing/k8s_tests.rst:
##########
@@ -358,7 +358,14 @@ Should show the status of current KinD cluster.
Cluster healthy: airflow-python-3.8-v1.24.2
5. Build the image base on PROD Airflow image. You need to build the PROD
image first (the command will
- guide you if you did not - either by running the build separately or
passing ``--rebuild-base-image`` flag
+ guide you if you did not - either by running the build separately or
passing ``--rebuild-base-image``
+ flag. Generally speaking you should not need to rebuuld the base image
unless you changed some
Review Comment:
```suggestion
flag. Generally speaking you should not need to rebuild the base image
unless you changed some
```
##########
docs/docker-stack/build.rst:
##########
@@ -432,6 +424,32 @@ The following example adds ``lxml`` python package from
PyPI to the image.
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]
+Example of adding ``PyPI`` package with constraints
+...................................................
+
+The following example adds ``lxml`` python package from PyPI to the image with
constraints that were
+used to install airflow. This allows you to use version of packages that you
know were tested with the
+given version of Airflow, so you can use it if you do not want to use
potentially newer versions
+that were released after the version of Airflow you are using.
Review Comment:
```suggestion
used to install airflow. This allows you to use the version of packages that
you know were tested with the
given version of Airflow. You can also use it if you do not want to use
potentially newer versions
that were released after the version of Airflow you are using.
```
##########
docs/docker-stack/build.rst:
##########
@@ -334,22 +334,27 @@ Naming conventions for the images:
Important notes for the base images
-----------------------------------
-You should be aware, about a few things:
+You should be aware, about a few things
* The production image of airflow uses "airflow" user, so if you want to add
some of the tools
as ``root`` user, you need to switch to it with ``USER`` directive of the
Dockerfile and switch back to
``airflow`` user when you are done. Also you should remember about following
the
`best practices of Dockerfiles
<https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_
to make sure your image is lean and small.
-* The PyPI dependencies in Apache Airflow are installed in the user library,
of the "airflow" user, so
- PIP packages are installed to ``~/.local`` folder as if the ``--user`` flag
was specified when running PIP.
- Note also that using ``--no-cache-dir`` is a good idea that can help to make
your image smaller.
+* You can use regular ``pip install`` commands (and as of Dockerfile coming in
Airflow 2.9 also
+ ``uv pip install`` (experimental) to install PyPI packages. Regular
``install`` commands should be used
Review Comment:
```suggestion
* You can use regular ``pip install`` commands (and as of Dockerfile coming
in Airflow 2.9 also
``uv pip install`` - experimental) to install PyPI packages. Regular
``install`` commands should be used,
```
##########
contributing-docs/testing/k8s_tests.rst:
##########
@@ -358,7 +358,14 @@ Should show the status of current KinD cluster.
Cluster healthy: airflow-python-3.8-v1.24.2
5. Build the image base on PROD Airflow image. You need to build the PROD
image first (the command will
- guide you if you did not - either by running the build separately or
passing ``--rebuild-base-image`` flag
+ guide you if you did not - either by running the build separately or
passing ``--rebuild-base-image``
Review Comment:
```suggestion
guide you if you did not) either by running the build separately or
passing ``--rebuild-base-image``
```
##########
Dockerfile.ci:
##########
@@ -415,23 +412,23 @@ function install_airflow_dependencies_from_branch_tip() {
set +x
common::install_packaging_tools
set -x
+ echo "${COLOR_BLUE}Uninstalling providers. Dependencies
remain${COLOR_RESET}"
# Uninstall airflow and providers to keep only the dependencies. In the
future when
# planned https://github.com/pypa/pip/issues/11440 is implemented in pip
we might be able to use this
# flag and skip the remove step.
- ${PACKAGING_TOOL_CMD} freeze | grep apache-airflow-providers | xargs
${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} 2>/dev/null || true
+ pip freeze | grep apache-airflow-providers | xargs ${PACKAGING_TOOL_CMD}
uninstall ${EXTRA_UNINSTALL_FLAGS} || true
Review Comment:
Why is this forced to be `pip` instead of `${PACKAGING_TOOL_CMD}`?
##########
docs/docker-stack/build.rst:
##########
@@ -432,6 +424,32 @@ The following example adds ``lxml`` python package from
PyPI to the image.
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]
+Example of adding ``PyPI`` package with constraints
+...................................................
+
+The following example adds ``lxml`` python package from PyPI to the image with
constraints that were
+used to install airflow. This allows you to use version of packages that you
know were tested with the
+given version of Airflow, so you can use it if you do not want to use
potentially newer versions
+that were released after the version of Airflow you are using.
+
+.. exampleinclude::
docker-examples/extending/add-pypi-packages-constraints/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Example of adding ``PyPI`` package with uv
+..........................................
+
+The following example adds ``lxml`` python package from PyPI to the image
using ``uv``. This is an
+experimental feature as ``uv`` is a very fast but also very new tool in Python
ecosystem.
+
Review Comment:
```suggestion
The following example adds ``lxml`` python package from PyPI to the image
using ``uv``. This is an
experimental feature as ``uv`` is a very fast but also very new tool in the
Python ecosystem.
```
##########
docs/docker-stack/build.rst:
##########
@@ -334,22 +334,27 @@ Naming conventions for the images:
Important notes for the base images
-----------------------------------
-You should be aware, about a few things:
+You should be aware, about a few things
* The production image of airflow uses "airflow" user, so if you want to add
some of the tools
as ``root`` user, you need to switch to it with ``USER`` directive of the
Dockerfile and switch back to
``airflow`` user when you are done. Also you should remember about following
the
`best practices of Dockerfiles
<https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_
to make sure your image is lean and small.
-* The PyPI dependencies in Apache Airflow are installed in the user library,
of the "airflow" user, so
- PIP packages are installed to ``~/.local`` folder as if the ``--user`` flag
was specified when running PIP.
- Note also that using ``--no-cache-dir`` is a good idea that can help to make
your image smaller.
+* You can use regular ``pip install`` commands (and as of Dockerfile coming in
Airflow 2.9 also
+ ``uv pip install`` (experimental) to install PyPI packages. Regular
``install`` commands should be used
+ however you should remember to add ``apache-airflow==${AIRFLOW_VERSION}`` to
the command to avoid
+ accidentally upgrading or downgrading the version of Apache Airflow. Depends
on the scenario you might
Review Comment:
```suggestion
accidentally upgrading or downgrading the version of Apache Airflow.
Depending on the scenario you might
```
##########
docs/docker-stack/build.rst:
##########
@@ -775,6 +793,19 @@ The following example builds the production image in
version ``3.8`` based on ``
:end-before: [END build]
+.. _image-build-uv:
+
+Building Using UV as package installer
+......................................
+
+The following example builds the production image in default settings, but
uses ``uv`` to build the image.
+This is experimental feature as ``uv`` is a very fast but also very new tool
in Python ecosystem.
Review Comment:
```suggestion
This is an experimental feature as ``uv`` is a very fast but also very new
tool in the Python ecosystem.
```
##########
docs/docker-stack/build.rst:
##########
@@ -775,6 +793,19 @@ The following example builds the production image in
version ``3.8`` based on ``
:end-before: [END build]
+.. _image-build-uv:
+
+Building Using UV as package installer
Review Comment:
```suggestion
Building prod images using UV as the package installer
```
##########
scripts/docker/install_airflow.sh:
##########
@@ -30,12 +35,21 @@
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
function install_airflow() {
- # Coherence check for editable installation mode.
- if [[ ${AIRFLOW_INSTALLATION_METHOD} != "." && \
- ${AIRFLOW_INSTALL_EDITABLE_FLAG} == "--editable" ]]; then
+ # Determine the installation_command_flags based on
AIRFLOW_INSTALLATION_METHOD method
Review Comment:
```suggestion
# Determine the installation_command_flags based on
AIRFLOW_INSTALLATION_METHOD method
```
##########
docs/docker-stack/changelog.rst:
##########
@@ -45,13 +45,25 @@ Airflow 2.9
``apache/airflow:slim-2.9.0-python-3.8`` images respectively so while the
change is potentially
breaking, it is very easy to switch to the previous behaviour.
-Airflow 2.9
-~~~~~~~~~~~
-
-The ``gosu`` binary was removed from the image. This is a potentially breaking
change for users who relied on
-``gosu`` to change the user in the container. The ``gosu`` binary was removed
because it was a source of
-security vulnerabilities as it was linked against older go standard libraries.
-
+ * The ``PIP_USER`` flag is removed and replaced by ``VIRTUAL_ENV`` pointing
to ``~/.local`` where Airflow
+ is installed. This has the effect that Airflow installation is treated as
regular virtual environment,
+ but unlike a regular virtualenv, the ``~/.local`` directory is seen as
``system level`` and when
+ worker creates dynamically virtualenv with ``--system-site-packages``
flag, Airflow installation and all
+ packages there are also present in the new virtualenv - but when you do
not use the flag, they are not
+ copied there which is backwards-compatible behaviour with having
``PIP_USER`` set.
+
+ * The image contains latest ``uv`` binary (latest at the moment of release)
- which is a new faster
+ replacement for ``pip``. While the image is still using ``pip`` by
default, you can use ``uv`` by default
+ to install packages and - experimentally - you can also build custom
images with
+ ``--arg AIRFLOW_USE_UV=true`` which will us ``uv`` to perform the
installation. This is an experimental
+ support, as ``uv`` is very fast but also very new feature in Python
ecosystem.
+
+ * Constraints used to install the image are available in
"${HOME}/constraints.txt" now - you can use them
+ to install additional packages in the image without having to find out
which constraints you should use.
+
+ * The ``gosu`` binary was removed from the image. This is a potentially
breaking change for users who relied on
+ ``gosu`` to change the user in the container. The ``gosu`` binary was
removed because it was a source of
+ security vulnerabilities as it was linked against older go standard
libraries.
Review Comment:
```suggestion
is installed. This has the effect that the Airflow installation is
treated as a regular virtual environment,
but unlike a regular virtualenv, the ``~/.local`` directory is seen as
``system level`` and when the
worker creates dynamically the virtualenv with
``--system-site-packages`` flag, the Airflow installation and all
packages there are also present in the new virtualenv. When you do not
use the flag, they are not
copied there which is a backwards-compatible behaviour with having
``PIP_USER`` set.
* The image contains latest ``uv`` binary (latest at the moment of
release) - which is a new faster
replacement for ``pip``. While the image is still using ``pip`` by
default, you can use ``uv``
to install packages and - experimentally - you can also build custom
images with
``--arg AIRFLOW_USE_UV=true`` which will us ``uv`` to perform the
installation. This is an experimental
support, as ``uv`` is very fast but also a very new feature in the
Python ecosystem.
* Constraints used to install the image are available in
"${HOME}/constraints.txt" now - you can use them
to install additional packages in the image without having to find out
which constraints you should use.
* The ``gosu`` binary was removed from the image. This is a potentially
breaking change for users who relied on
``gosu`` to change the user in the container. The ``gosu`` binary was
removed because it was a source of
security vulnerabilities as it was linked against older Go standard
libraries.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
