This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eed7d335982404d3cda12f587799f2da4b176bc2 Author: Jarek Potiuk <[email protected]> AuthorDate: Tue Jun 2 10:24:47 2020 +0200 You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch. (cherry picked from commit a6216a760ca7449cfbf5aeee0f74caeda2f3f999) --- .github/workflows/ci.yml | 2 +- BREEZE.rst | 148 ++++++++++++++++++------- breeze | 106 +++++++++++++----- breeze-complete | 13 ++- scripts/ci/_utils.sh | 35 +++--- scripts/ci/ci_fix_ownership.sh | 9 ++ scripts/ci/in_container/_in_container_utils.sh | 2 +- 7 files changed, 234 insertions(+), 81 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e4f16e7..afa911c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,7 +32,7 @@ env: VERBOSE: "true" UPGRADE_TO_LATEST_REQUIREMENTS: "false" PYTHON_MAJOR_MINOR_VERSION: 3.5 - CACHE_REGISTRY: "docker.pkg.github.com" + ENABLE_REGISTRY_CACHE: "true" CACHE_IMAGE_PREFIX: ${{ github.repository }} CACHE_REGISTRY_USERNAME: ${{ github.actor }} CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} diff --git a/BREEZE.rst b/BREEZE.rst index 262ddcd..0d90763 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -78,36 +78,46 @@ Docker Compose - **Permissions**: Configure to run the ``docker-compose`` command. -Docker in WSL -------------- +Docker in WSL 2 +--------------- + +- **WSL 2 installation** : + Install WSL 2 and a Linux Distro (e.g. Ubuntu) see + `WSL 2 Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details. -- **WSL installation** : - `WSL Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details. +- **Docker Desktop installation** : + Install Docker Desktop for Windows. For Windows Home follow the + `Docker Windows Home Installation Guide <https://docs.docker.com/docker-for-windows/install-windows-home>`_. + For Windows Pro, Enterprise, or Education follow the + `Docker Windows Installation Guide <https://docs.docker.com/docker-for-windows/install/>`_. -- **Docker installation** : - You should install docker in WSL. - follow `Docker Installtion Guide <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`_ - only docker-ce without docker-ce-cli containerd.io. - **Docker setting** : - You should expose Docker daemon, + WSL integration needs to be enabled -.. image:: images/docker_expose_daemon.png +.. image:: images/docker_wsl_integration.png :align: left - :alt: Docker expose daemon - -and set env variable DOCKER_HOST. - -.. code-block:: bash - - echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc - -- **WSL problems** : - There is a mounting problem in docker because docker could not recognize ``/mnt/c``, ``/mnt/d`` driver path. - run this command in Windows Version 18.03+ and reboot Windows - -.. code-block:: bash - - printf '[automount]\nroot = /\n options = "metadata"\n' >> /etc/wsl.conf + :alt: Docker WSL2 integration + +- **WSL 2 Filesystem Performance** : + Accessing the host Windows filesystem incurs a performance penalty, + it is therefore recommended to do development on the Linux filesystem. + E.g. Run ``cd ~`` and create a development folder in your Linux distro home + and git pull the Airflow repo there. + +- **WSL 2 Memory Usage** : + WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim + the memory after development you can: + * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3`` + * If no longer using Docker you can quit Docker Desktop + (right click system try icon and select "Quit Docker Desktop") + * If no longer using WSL you can shut it down on the Windows Host + with the following command: ``wsl --shutdown`` + +- **Developing in WSL 2** : + You can use all the standard Linux command line utilities to develop on WSL 2. + Further VS Code supports developing in Windows but remotely executing in WSL. + If VS Code is installed on the Windows host system then in the WSL Linux Distro + you can run ``code .`` in the root directory of you Airflow repo to launch VS Code. Docker Images Used by Breeze ---------------------------- @@ -182,6 +192,8 @@ On macOS, 2GB of RAM are available for your Docker containers by default, but mo (4GB should be comfortable). For details see `Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_. +On Windows WSL 2 expect the Linux Disto and Docker containers to use 7 - 8 GB of RAM. + Airflow Directory Structure inside Docker ----------------------------------------- @@ -655,6 +667,7 @@ This is the current syntax for `./breeze <./breeze>`_: cleanup-image Cleans up the container image created exec Execs into running breeze container in new terminal generate-requirements Generates pinned requirements for pip dependencies + push-image Pushes images to registry initialize-local-virtualenv Initializes local virtualenv setup-autocomplete Sets up autocomplete for breeze stop Stops the docker-compose environment @@ -729,7 +742,7 @@ This is the current syntax for `./breeze <./breeze>`_: breeze [FLAGS] build-image -- <EXTRA_ARGS> Builds docker image (CI or production) without entering the container. You can pass - aditional options to this command, such as '--force-build-image', + additional options to this command, such as '--force-build-image', '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour. You can also pass '--production-image' flag to build production image rather than CI image. @@ -788,17 +801,23 @@ This is the current syntax for `./breeze <./breeze>`_: Uses local cache to build images. No pulled images will be used, but results of local builds in the Docker cache are used instead. - -u, --push-images - After building - uploads the images to DockerHub - It is useful in case you use your own DockerHub user to store images and you want - to build them locally. Note that you need to use 'docker login' before you upload images. - -D, --dockerhub-user DockerHub user used to pull, push and build images. Default: apache. -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: airflow. + -c, --registry-cache + If registry cache is enabled, pulls and pushes are done from the registry cache in github. + You need to be logged in to the registry in order to be able to pull/push from it and you + need to be committer to push to airflow registry. + + -G, --github-organisation + GitHub organisation used to pull, push images when cache is used. Default: apache. + + -g, --github-repo + GitHub repository used to pull, push images when cache is used. Default: airflow. + -v, --verbose Show verbose information about executed commands (enabled by default for running test). Note that you can further increase verbosity and see all the commands executed by breeze @@ -876,6 +895,55 @@ This is the current syntax for `./breeze <./breeze>`_: #################################################################################################### + Detailed usage for command: push-image + + breeze [FLAGS] push-image -- <EXTRA_ARGS> + + Pushes images to docker registry. You can push the images to DockerHub registry (default) + or to the GitHub cache registry (if --registry-cache flag is used). + + For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify + the repository to push to. For GitHub repository --github-organisation and --github-repo + flags can be used for the same purpose. + + You can also add --production-image flag to switch to production image (default is CI one) + + Examples: + + 'breeze push-image' or + 'breeze push-image --dockerhub-user user' to push to your private registry or + 'breeze push-image --production-image' - to push production image or + 'breeze push-image --registry-cache' - to push to GitHub cache or + 'breeze push-image --registry-cache --github-organisation org' - for other organisation + + Flags: + + -D, --dockerhub-user + DockerHub user used to pull, push and build images. Default: apache. + + -H, --dockerhub-repo + DockerHub repository used to pull, push, build images. Default: airflow. + + -c, --registry-cache + If registry cache is enabled, pulls and pushes are done from the registry cache in github. + You need to be logged in to the registry in order to be able to pull/push from it and you + need to be committer to push to airflow registry. + + -G, --github-organisation + GitHub organisation used to pull, push images when cache is used. Default: apache. + + -g, --github-repo + GitHub repository used to pull, push images when cache is used. Default: airflow. + + -v, --verbose + Show verbose information about executed commands (enabled by default for running test). + Note that you can further increase verbosity and see all the commands executed by breeze + by running 'export VERBOSE_COMMANDS="true"' before running breeze. + + + #################################################################################################### + + Detailed usage for command: initialize-local-virtualenv breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS> @@ -1333,12 +1401,7 @@ This is the current syntax for `./breeze <./breeze>`_: builds in the Docker cache are used instead. **************************************************************************************************** - Flags for pushing Docker images (both CI and production) - - -u, --push-images - After building - uploads the images to DockerHub - It is useful in case you use your own DockerHub user to store images and you want - to build them locally. Note that you need to use 'docker login' before you upload images. + Flags for pulling/pushing Docker images (both CI and production) -D, --dockerhub-user DockerHub user used to pull, push and build images. Default: apache. @@ -1346,6 +1409,17 @@ This is the current syntax for `./breeze <./breeze>`_: -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: airflow. + -c, --registry-cache + If registry cache is enabled, pulls and pushes are done from the registry cache in github. + You need to be logged in to the registry in order to be able to pull/push from it and you + need to be committer to push to airflow registry. + + -G, --github-organisation + GitHub organisation used to pull, push images when cache is used. Default: apache. + + -g, --github-repo + GitHub repository used to pull, push images when cache is used. Default: airflow. + **************************************************************************************************** Increase verbosity of the scripts diff --git a/breeze b/breeze index 75c00eb..fcea655 100755 --- a/breeze +++ b/breeze @@ -67,7 +67,7 @@ function setup_default_breeze_variables() { # We have different versions of images depending on the python version used. We keep up with the # Latest patch-level changes in Python (this is done automatically during CI builds) so we have - # To only take into account MAJOR and MINOR version of python. This variable keeps the major/mninor + # To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor # version of python in X.Y format (3.6, 3.7 etc). export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}" @@ -158,7 +158,7 @@ function setup_default_breeze_variables() { # Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary # packages from PyPI and it case of problems it provides useful hints on what prerequisites should be -# installed. It also removes and resets the existing AIRFOW_HOME installation to make sure that you +# installed. It also removes and resets the existing AIRFLOW_HOME installation to make sure that you # have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to # a clean state. You can use this function if your virtualenv is broken, to clean it up function initialize_virtualenv() { @@ -792,13 +792,24 @@ function parse_arguments() { echo export FORWARD_CREDENTIALS="true" shift 1 ;; - -u|--push-images) + -c|--registry-cache) echo - echo "Pushing images to DockerHub" + echo "Use cache for the container registry" echo - export PUSH_IMAGES="true" - export FORCE_BUILD_IMAGES="true" + export ENABLE_REGISTRY_CACHE="true" shift ;; + -G|--github-organisation) + echo + echo "GitHub organisation" + echo + export GITHUB_ORGANISATION="${2}" + shift 2;; + -g|--github-repo) + echo + echo "GitHub repository" + echo + export GITHUB_REPOSITORY="${2}" + shift 2;; --) shift ; break ;; @@ -860,6 +871,15 @@ function parse_arguments() { generate-requirements) LAST_SUBCOMMAND="${1}" COMMAND_TO_RUN="perform_generate_requirements" + # if you want to generate requirement - you want to build the image too :) + export FORCE_ANSWER_TO_QUESTIONS="yes" + # and assume you want to build it no matter if it is needed + export FORCE_BUILD_IMAGES="true" + shift ;; + push-image) + LAST_SUBCOMMAND="${1}" + COMMAND_TO_RUN="perform_push_image" + SKIP_CHECK_REMOTE_IMAGE="true" shift ;; initialize-local-virtualenv) LAST_SUBCOMMAND="${1}" @@ -1050,6 +1070,7 @@ function prepare_usage() { export USAGE_FLAGS="Shows all breeze's flags" export USAGE_GENERATE_REQUIREMENTS="Generates pinned requirements for pip dependencies" export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv" + export USAGE_PUSH_IMAGE="Pushes images to registry" export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze" export USAGE_STOP="Stops the docker-compose environment" export USAGE_RESTART="Stops the docker-compose environment including DB cleanup" @@ -1100,7 +1121,7 @@ $(flag_footer) # shellcheck disable=SC2089 DETAILED_USAGE_BUILD_IMAGE=" Builds docker image (CI or production) without entering the container. You can pass - aditional options to this command, such as '--force-build-image', + additional options to this command, such as '--force-build-image', '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour. You can also pass '--production-image' flag to build production image rather than CI image. @@ -1109,7 +1130,7 @@ $(flag_airflow_variants) $(flag_choose_different_airflow_version) $(flag_production_image) $(flag_build_docker_images) -$(flag_push_docker_images) +$(flag_pull_push_docker_images) $(flag_verbosity) " # shellcheck disable=SC2090 @@ -1187,6 +1208,31 @@ $(flag_airflow_variants) " # shellcheck disable=SC2090 export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV + # shellcheck disable=SC2089 + DETAILED_USAGE_PUSH_IMAGE=" + Pushes images to docker registry. You can push the images to DockerHub registry (default) + or to the GitHub cache registry (if --registry-cache flag is used). + + For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify + the repository to push to. For GitHub repository --github-organisation and --github-repo + flags can be used for the same purpose. + + You can also add --production-image flag to switch to production image (default is CI one) + + Examples: + + '${CMDNAME} push-image' or + '${CMDNAME} push-image --dockerhub-user user' to push to your private registry or + '${CMDNAME} push-image --production-image' - to push production image or + '${CMDNAME} push-image --registry-cache' - to push to GitHub cache or + '${CMDNAME} push-image --registry-cache --github-organisation org' - for other organisation + +Flags: +$(flag_pull_push_docker_images) +$(flag_verbosity) +" + # shellcheck disable=SC2090 + export DETAILED_USAGE_PUSH_IMAGE export DETAILED_USAGE_SETUP_AUTOCOMPLETE=" Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash shell and when typing breeze command <TAB> will provide autocomplete for @@ -1548,19 +1594,26 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS} " } -# Prints flags controlling docker push process -function flag_push_docker_images() { +# Prints flags controlling docker pull and push process +function flag_pull_push_docker_images() { echo " --u, --push-images - After building - uploads the images to DockerHub - It is useful in case you use your own DockerHub user to store images and you want - to build them locally. Note that you need to use 'docker login' before you upload images. - -D, --dockerhub-user DockerHub user used to pull, push and build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_USER:=}. -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_REPO:=}. + +-c, --registry-cache + If registry cache is enabled, pulls and pushes are done from the registry cache in github. + You need to be logged in to the registry in order to be able to pull/push from it and you + need to be committer to push to airflow registry. + +-G, --github-organisation + GitHub organisation used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_ORGANISATION:=}. + +-g, --github-repo + GitHub repository used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_REPO:=}. + " } @@ -1613,8 +1666,8 @@ $(print_star_line) $(flag_build_docker_images) $(print_star_line) - Flags for pushing Docker images (both CI and production) -$(flag_push_docker_images) + Flags for pulling/pushing Docker images (both CI and production) +$(flag_pull_push_docker_images) $(print_star_line) Increase verbosity of the scripts @@ -1822,7 +1875,7 @@ function run_static_checks { fi } -# Runs Build before a comman - it will check and if needed rebuild necessary image, depending on the +# Runs Build before a command - it will check and if needed rebuild necessary image, depending on the # command chosen function run_build_command { case "${COMMAND_TO_RUN}" in @@ -1838,7 +1891,7 @@ function run_build_command { rebuild_ci_image_if_needed fi ;; - build_docs|perform_static_checks|perform_generate_requirements) + build_docs|perform_static_checks|perform_generate_requirements|perform_push_image) prepare_ci_build rebuild_ci_image_if_needed ;; @@ -1866,6 +1919,7 @@ function run_build_command { echo >&2 echo >&2 "ERROR: Unknown command to run ${COMMAND_TO_RUN}" echo >&2 + exit 1 ;; esac } @@ -1923,13 +1977,6 @@ function run_breeze_command { run_static_checks "$@" ;; build_image) - if [[ ${PUSH_IMAGES} == "true" ]]; then - if [[ ${PRODUCTION_IMAGE} == "true" ]]; then - push_prod_images - else - push_ci_image - fi - fi ;; cleanup_image) remove_images @@ -1937,6 +1984,13 @@ function run_breeze_command { perform_generate_requirements) run_generate_requirements ;; + perform_push_image) + if [[ ${PRODUCTION_IMAGE} == "true" ]]; then + push_prod_images + else + push_ci_image + fi + ;; perform_initialize_local_virtualenv) initialize_virtualenv ;; diff --git a/breeze-complete b/breeze-complete index 8afe550..de39a83 100644 --- a/breeze-complete +++ b/breeze-complete @@ -79,6 +79,8 @@ EOF _BREEZE_DEFAULT_DOCKERHUB_USER="apache" _BREEZE_DEFAULT_DOCKERHUB_REPO="airflow" +_BREEZE_DEFAULT_GITHUB_ORGANISATION="apache" +_BREEZE_DEFAULT_GITHUB_REPO="airflow" _BREEZE_SHORT_OPTIONS=" h p: b: i: @@ -87,7 +89,7 @@ s x r l a: t: d: v y n q f F P I E: C L -D: R: u +D: R: c g: G: " _BREEZE_LONG_OPTIONS=" @@ -97,7 +99,7 @@ kind-cluster-start kind-cluster-stop kind-cluster-recreate skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset verbose assume-yes assume-no assume-quit forward-credentials force-build-images force-pull-images production-image extras: force-clean-images use-local-cache -dockerhub-user: dockerhub-repo: push-images +dockerhub-user: dockerhub-repo: registry-cache github-organisation: github-repo: postgres-version: mysql-version: additional-extras: additional-python-deps: " @@ -109,6 +111,7 @@ build-image cleanup-image exec generate-requirements +push-image initialize-local-virtualenv setup-autocomplete stop @@ -179,6 +182,12 @@ function _get_known_values_breeze() { -R | --dockerhub-repo) _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_DOCKERHUB_REPO}" ;; + -G | --github-organisation) + _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_ORGANISATION}" + ;; + -g | --github-repo) + _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_REPO}" + ;; *) _BREEZE_KNOWN_VALUES="" ;; diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh index 6c0204c..4ba7d83 100644 --- a/scripts/ci/_utils.sh +++ b/scripts/ci/_utils.sh @@ -83,6 +83,11 @@ function initialize_common_environment { # Default branch name for triggered builds is the one configured in default branch export BRANCH_NAME=${BRANCH_NAME:=${DEFAULT_BRANCH}} + export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"} + export GITHUB_REPO=${GITHUB_REPO:="airflow"} + export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"} + export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"} + # Default port numbers for forwarded ports export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"} export POSTGRES_HOST_PORT=${POSTGRES_HOST_PORT:="25433"} @@ -92,10 +97,6 @@ function initialize_common_environment { export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"} export MYSQL_VERSION=${MYSQL_VERSION:="5.6"} - # Do not push images by default (push them directly from the build script on Dockerhub or when - # --push-images flag is specified - export PUSH_IMAGES=${PUSH_IMAGES:="false"} - # Whether base python images should be pulled from cache export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} @@ -1455,11 +1456,14 @@ function prepare_ci_build() { export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest" export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest" export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}" - if [[ ${CACHE_REGISTRY=""} != "" ]]; then - echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ - --username "${CACHE_REGISTRY_USERNAME}" \ - --password-stdin \ - "${CACHE_REGISTRY}" + if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then + if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then + echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ + --username "${CACHE_REGISTRY_USERNAME}" \ + --password-stdin \ + "${CACHE_REGISTRY}" + fi + export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}} export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}" export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster" else @@ -1543,11 +1547,14 @@ function prepare_prod_build() { export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}" export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}" - if [[ ${CACHE_REGISTRY=""} != "" ]]; then - echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ - --username "${CACHE_REGISTRY_USERNAME}" \ - --password-stdin \ - "${CACHE_REGISTRY}" + if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then + if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then + echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ + --username "${CACHE_REGISTRY_USERNAME}" \ + --password-stdin \ + "${CACHE_REGISTRY}" + fi + export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}} export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}" export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build" export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster" diff --git a/scripts/ci/ci_fix_ownership.sh b/scripts/ci/ci_fix_ownership.sh index b47d83a..775cc58 100755 --- a/scripts/ci/ci_fix_ownership.sh +++ b/scripts/ci/ci_fix_ownership.sh @@ -19,6 +19,7 @@ # # Fixes ownership for files created inside container (files owned by root will be owned by host user) # +set -x export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5} @@ -28,6 +29,14 @@ export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5} export AIRFLOW_CI_IMAGE=\ ${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci +export AIRFLOW_IMAGE=${AIRFLOW_CI_IMAGE} +export WEBSERVER_HOST_PORT=28080 +HOST_USER_ID="$(id -ur)" +HOST_GROUP_ID="$(id -gr)" + +export HOST_USER_ID +export HOST_GROUP_ID + docker-compose \ -f "${MY_DIR}/docker-compose/base.yml" \ -f "${MY_DIR}/docker-compose/local.yml" \ diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh index be386e0..9b9814a 100644 --- a/scripts/ci/in_container/_in_container_utils.sh +++ b/scripts/ci/in_container/_in_container_utils.sh @@ -93,7 +93,7 @@ function in_container_cleanup_pycache() { # function in_container_fix_ownership() { set +o pipefail - sudo find . -user root -print0 \ + sudo find "${AIRFLOW_SOURCES}" -print0 -user root \ | sudo xargs --null chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1 set -o pipefail }
