This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new d8a42cadf8 Improve speed of regular upgrades when dependencies change.
(#37360)
d8a42cadf8 is described below
commit d8a42cadf85df026f4e41e5e440e350380615f4a
Author: Jarek Potiuk <[email protected]>
AuthorDate: Mon Feb 12 21:30:54 2024 +0100
Improve speed of regular upgrades when dependencies change. (#37360)
When regular dependencies change we enable UPGRADE_TO_NEWER_DEPENDENCIES
flag. After some recent improvements in installing dependencies from
the branch tip where we can install airflow using `[devel-ci]` extra
from the branch tip, this opens up another option for speeding up
the installations, when some dependencies change.
So far - UPGRADE_TO_NEWER_DEPENDENCIES necessitated to reinstall all
dependencies with airflow attempting to upgrade them eagerly. This
takes ~ 28 minutes on CI now from the scratch. However, this is not
needed. We can still start from the set of dependencies pre-installed
by installation from branch tip, and only trigger
"UPGRADE_TO_NEWER_DEPENDENCIES" after that - so that the "URL"
installation cache does not get invalidated.
This should limit the time needed to build such an image to ~ 8 minutes
in CI when cache is built.
In the future we might further optimize it - as we now have all the
mechanisms necessary to do that in selective checks. For example we
could trigger UPGRADE_TO_NEWER_DEPENDENCIES only when some dependencies
are changes, and not when new dependencies are added, further limiting
the time needed to build such image to some ~ 3/4 minutes. But this
should be a separate PR and should be carefuly thought about, as
it might also be that this will lead to conflicts with the latest
constraints..
---
Dockerfile.ci | 37 ++++++++++++++++++-------------------
1 file changed, 18 insertions(+), 19 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 4b6968f096..6d75ac2d51 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1127,23 +1127,6 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
RUN echo "Airflow version: ${AIRFLOW_VERSION}"
-# Those are additional constraints that are needed for some extras but we do
not want to
-# force them on the main Airflow package. Currently we need no extra limits as
PIP 23.1+ has much better
-# dependency resolution and we do not need to limit the versions of the
dependencies
-#
-# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33
and we want to help
-# `pip` to limit the versions it checks and limit backtracking, by explicitly
specifying these limits
-# when performing eager upgrade of dependencies - this way it won't even
consider 1.34 versions of boto
-# We should update it every time a new version of aiobotocore is released
supporting 1.34
-#
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"
-ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
-ARG VERSION_SUFFIX_FOR_PYPI=""
-
-ENV
EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
- UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} \
- VERSION_SUFFIX_FOR_PYPI=${VERSION_SUFFIX_FOR_PYPI}
-
# Copy all scripts required for installation - changing any of those should
lead to
# rebuilding from here
COPY --from=scripts install_pip_version.sh
install_airflow_dependencies_from_branch_tip.sh \
@@ -1158,8 +1141,7 @@ COPY --from=scripts install_pip_version.sh
install_airflow_dependencies_from_bra
# the cache is only used when "upgrade to newer dependencies" is not set to
automatically
# account for removed dependencies (we do not install them in the first place)
RUN bash /scripts/docker/install_pip_version.sh; \
- if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
- ${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then \
+ if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
fi
@@ -1184,6 +1166,23 @@ COPY airflow_pre_installed_providers.txt
${AIRFLOW_SOURCES}/
COPY hatch_build.py ${AIRFLOW_SOURCES}/
COPY --from=scripts install_airflow.sh /scripts/docker/
+# Those are additional constraints that are needed for some extras but we do
not want to
+# force them on the main Airflow package. Currently we need no extra limits as
PIP 23.1+ has much better
+# dependency resolution and we do not need to limit the versions of the
dependencies
+#
+# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33
and we want to help
+# `pip` to limit the versions it checks and limit backtracking, by explicitly
specifying these limits
+# when performing eager upgrade of dependencies - this way it won't even
consider 1.34 versions of boto
+# We should update it every time a new version of aiobotocore is released
supporting 1.34
+#
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+ARG VERSION_SUFFIX_FOR_PYPI=""
+
+ENV
EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
+ UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} \
+ VERSION_SUFFIX_FOR_PYPI=${VERSION_SUFFIX_FOR_PYPI}
+
# The goal of this line is to install the dependencies from the most current
pyproject.toml from sources
# This will be usually incremental small set of packages in CI optimized
build, so it will be very fast
# In non-CI optimized build this will install all dependencies before
installing sources.
