potiuk commented on a change in pull request #14107:
URL: https://github.com/apache/airflow/pull/14107#discussion_r571138631
##########
File path: scripts/ci/images/ci_build_dockerhub.sh
##########
@@ -113,8 +113,8 @@ else
export DOCKER_CACHE="local"
# Name the image based on the TAG rather than based on the branch name
export FORCE_AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG}"
- export INSTALL_AIRFLOW_VERSION="${DOCKER_TAG%-python*}"
- export
AIRFLOW_CONSTRAINTS_REFERENCE="constraints-${INSTALL_AIRFLOW_VERSION}"
+ export INSTALL_AIRFLOW_VERSION="==${DOCKER_TAG%-python*}"
Review comment:
> I think what I am failing to understand is if
`AIRFLOW_INSTALL_VERSION` needs `==VERSION` or `<=VERSION` etc. How does
>
>
https://github.com/apache/airflow/blob/fc67521f31a0c9a74dadda8d5f0ac32c07be218d/.github/workflows/ci.yml#L489
>
> work as it looks like `"--build-arg"
"AIRFLOW_INSTALL_VERSION=${INSTALL_AIRFLOW_VERSION}"` it is passed verbatim
Yes because that only affects the way how PROD image is built. For CI image
it is different - we always built it from sources, but then when we enter the
CI Image we can remove/reinstall airflow using either a PyPI version or locally
built wheel/sdist. This allows to use the same CI image to test different
airflow version (this is how we are testing backport providers - we take the CI
image, reinstall airflow to 1.10.14 from pypi and then install all backport
providers built locally.
That's what `INSTALL_AIRFLOW_VERSION: "1.10.14"` line in ci.yml is.
Similarly we take our providers and install them in 2.0.0 version as there was
a change in get_provider_info() schema and we want to make sure that the
providers install in 2.0.0 version as well. This is is what
`INSTALL_AIRFLOW_VERSION: "2.0.0"` does in ci.yml.
Also we use it to install airflow from wheels or sdist packages provided
`INSTALL_AIRFLOW_VERSION: "${{ matrix.package-format }}"` -> in this case we
are simply using `wheel` or `sdist` there.
Here is the specification from breeze explaining it:
```
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
If specified, installs Airflow directly from PIP released version.
This happens at
image building time in production image and at container entering
time for CI image. One of:
2.0.0 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel
sdist
When 'none' is used, you can install airflow from local packages.
When building image,
airflow package should be added to 'docker-context-files' and
--install-from-docker-context-files flag should be used. When
running an image, airflow
package should be added to dist folder and
--install-packages-from-dist flag should be used.
```
If you look at entrypoint_ci.sh this piece of code is exected in
entrypoint_ci.sh:
```
if [[ -z ${INSTALL_AIRFLOW_VERSION=} ]]; then
export PYTHONPATH=${AIRFLOW_SOURCES}
echo
echo "Using already installed airflow version"
echo
"${AIRFLOW_SOURCES}/airflow/www/ask_for_recompile_assets_if_needed.sh"
# Cleanup the logs, tmp when entering the environment
sudo rm -rf "${AIRFLOW_SOURCES}"/logs/*
sudo rm -rf "${AIRFLOW_SOURCES}"/tmp/*
mkdir -p "${AIRFLOW_SOURCES}"/logs/
mkdir -p "${AIRFLOW_SOURCES}"/tmp/
elif [[ ${INSTALL_AIRFLOW_VERSION} == "none" ]]; then
echo
echo "Skip installing airflow - only install wheel/tar.gz packages that
are present locally"
echo
uninstall_airflow_and_providers
elif [[ ${INSTALL_AIRFLOW_VERSION} == "wheel" ]]; then
echo
echo "Install airflow from wheel package with [${AIRFLOW_EXTRAS}] extras
but uninstalling providers."
echo
uninstall_airflow_and_providers
install_airflow_from_wheel "[${AIRFLOW_EXTRAS}]"
uninstall_providers
elif [[ ${INSTALL_AIRFLOW_VERSION} == "sdist" ]]; then
echo
echo "Install airflow from sdist package with [${AIRFLOW_EXTRAS}] extras
but uninstalling providers."
echo
uninstall_airflow_and_providers
install_airflow_from_sdist "[${AIRFLOW_EXTRAS}]"
uninstall_providers
else
echo
echo "Install airflow from PyPI including [${AIRFLOW_EXTRAS}] extras"
echo
install_released_airflow_version "${INSTALL_AIRFLOW_VERSION}"
"[${AIRFLOW_EXTRAS}]"
fi
```
We might want to change that varialle name in CI image to avoid confusion -
maybe REINSTALL_AIRFLOW_IN_CI ? - then we could make two parameters in breeze
instead of the shared one:
* `--install-airflow-version 2.0.0 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9`
for PROD image
* `--reinstall-airflow-in-ci 2.0.0 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9
none wheel sdist` for CI image
WDYT ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
