This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e0154561fd Add a possibility to remove the dependency from cache in a
PR (#38312)
e0154561fd is described below
commit e0154561fda4f2a04c25f82708df999e66dc8a5b
Author: Jarek Potiuk <[email protected]>
AuthorDate: Wed Mar 20 16:32:16 2024 +0100
Add a possibility to remove the dependency from cache in a PR (#38312)
When you rmeove a dependency from pyproject.toml, until your PR
gets merged, it is still present in the main branch of Airflow and
this is where we do a "warm" cache install of dependencies when
we build the CI image. So the dependency is stil there when your PR
is running tests. In order to remove the dependency for the time
of PR, we need to remove it manually in the script that installs
dependencies from branch tip (and also changing this script will
trigger Docker cache invalidation of the branch tip installation
step).
This PR adds ability to specify the package you remove to also be
removed after "branch tip" installation. After merging your PR,
it should be fine to remove it again from the branch tip.
---
Dockerfile | 20 ++++++++++++++++++++
Dockerfile.ci | 20 ++++++++++++++++++++
pyproject.toml | 4 ++++
.../install_airflow_dependencies_from_branch_tip.sh | 20 ++++++++++++++++++++
4 files changed, 64 insertions(+)
diff --git a/Dockerfile b/Dockerfile
index 9e207938b8..8355d61144 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -477,6 +477,26 @@ function install_airflow_dependencies_from_branch_tip() {
set +x
${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} apache-airflow
set -x
+ # If you want to make sure dependency is removed from cache in your PR
when you removed it from
+ # pyproject.toml - please add your dependency here as a list of strings
+ # for example:
+ # DEPENDENCIES_TO_REMOVE=("package_a" "package_b")
+ # Once your PR is merged, you should make a follow-up PR to remove it from
this list
+ # and increase the AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci to make sure
your cache is rebuilt.
+ local DEPENDENCIES_TO_REMOVE
+ # IMPORTANT!! Make sure to increase AIRFLOW_CI_BUILD_EPOCH in
Dockerfile.ci when you remove a dependency from that list
+ DEPENDENCIES_TO_REMOVE=()
+ if [[ "${DEPENDENCIES_TO_REMOVE[*]}" != "" ]]; then
+ echo
+ echo "${COLOR_BLUE}Uninstalling just removed dependencies (temporary
until cache refreshes)${COLOR_RESET}"
+ echo "${COLOR_BLUE}Dependencies to uninstall:
${DEPENDENCIES_TO_REMOVE[*]}${COLOR_RESET}"
+ echo
+ set +x
+ ${PACKAGING_TOOL_CMD} uninstall "${DEPENDENCIES_TO_REMOVE[@]}" || true
+ set -x
+ # make sure that the dependency is not needed by something else
+ pip check
+ fi
}
common::get_colors
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 992d3c26ed..7ea8e62db4 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -424,6 +424,26 @@ function install_airflow_dependencies_from_branch_tip() {
set +x
${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} apache-airflow
set -x
+ # If you want to make sure dependency is removed from cache in your PR
when you removed it from
+ # pyproject.toml - please add your dependency here as a list of strings
+ # for example:
+ # DEPENDENCIES_TO_REMOVE=("package_a" "package_b")
+ # Once your PR is merged, you should make a follow-up PR to remove it from
this list
+ # and increase the AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci to make sure
your cache is rebuilt.
+ local DEPENDENCIES_TO_REMOVE
+ # IMPORTANT!! Make sure to increase AIRFLOW_CI_BUILD_EPOCH in
Dockerfile.ci when you remove a dependency from that list
+ DEPENDENCIES_TO_REMOVE=()
+ if [[ "${DEPENDENCIES_TO_REMOVE[*]}" != "" ]]; then
+ echo
+ echo "${COLOR_BLUE}Uninstalling just removed dependencies (temporary
until cache refreshes)${COLOR_RESET}"
+ echo "${COLOR_BLUE}Dependencies to uninstall:
${DEPENDENCIES_TO_REMOVE[*]}${COLOR_RESET}"
+ echo
+ set +x
+ ${PACKAGING_TOOL_CMD} uninstall "${DEPENDENCIES_TO_REMOVE[@]}" || true
+ set -x
+ # make sure that the dependency is not needed by something else
+ pip check
+ fi
}
common::get_colors
diff --git a/pyproject.toml b/pyproject.toml
index 01d554e764..d8c63ba595 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -63,6 +63,10 @@ classifiers = [
"Programming Language :: Python :: 3.11",
"Topic :: System :: Monitoring",
]
+# When you remove a dependency from the list, you should also make sure to add
the dependency to be removed
+# in the scripts/docker/install_airflow_dependencies_from_branch_tip.sh script
DEPENDENCIES_TO_REMOVE
+# in order to make sure the dependency is not installed in the CI image build
process from the main
+# of Airflow branch. After your PR is merged, you should remove it from the
list there.
dependencies = [
# Alembic is important to handle our migrations in predictable and
performant way. It is developed
# together with SQLAlchemy. Our experience with Alembic is that it very
stable in minor version
diff --git a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
index f9c76f091a..8158ab5886 100644
--- a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
+++ b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
@@ -67,6 +67,26 @@ function install_airflow_dependencies_from_branch_tip() {
set +x
${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} apache-airflow
set -x
+ # If you want to make sure dependency is removed from cache in your PR
when you removed it from
+ # pyproject.toml - please add your dependency here as a list of strings
+ # for example:
+ # DEPENDENCIES_TO_REMOVE=("package_a" "package_b")
+ # Once your PR is merged, you should make a follow-up PR to remove it from
this list
+ # and increase the AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci to make sure
your cache is rebuilt.
+ local DEPENDENCIES_TO_REMOVE
+ # IMPORTANT!! Make sure to increase AIRFLOW_CI_BUILD_EPOCH in
Dockerfile.ci when you remove a dependency from that list
+ DEPENDENCIES_TO_REMOVE=()
+ if [[ "${DEPENDENCIES_TO_REMOVE[*]}" != "" ]]; then
+ echo
+ echo "${COLOR_BLUE}Uninstalling just removed dependencies (temporary
until cache refreshes)${COLOR_RESET}"
+ echo "${COLOR_BLUE}Dependencies to uninstall:
${DEPENDENCIES_TO_REMOVE[*]}${COLOR_RESET}"
+ echo
+ set +x
+ ${PACKAGING_TOOL_CMD} uninstall "${DEPENDENCIES_TO_REMOVE[@]}" || true
+ set -x
+ # make sure that the dependency is not needed by something else
+ pip check
+ fi
}
common::get_colors
