This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c7613c597a4bf67114423d2b68c451dd6719191e Author: Jarek Potiuk <[email protected]> AuthorDate: Tue Jun 16 12:36:46 2020 +0200 Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images. (cherry picked from commit 7c12a9d4e0b6c1e01fee6ab227a6e25b5aa5b157) --- .dockerignore | 2 +- .github/workflows/ci.yml | 2 +- BREEZE.rst | 219 +++---------- Dockerfile | 34 +- Dockerfile.ci | 17 +- IMAGES.rst | 346 +++++++++++++++++---- TESTING.rst | 10 +- breeze | 216 ++++--------- breeze-complete | 7 +- scripts/ci/ci_run_airflow_testing.sh | 6 +- scripts/ci/docker-compose/base.yml | 7 - scripts/ci/docker-compose/local-prod.yml | 2 +- scripts/ci/docker-compose/local.yml | 4 +- scripts/ci/in_container/entrypoint_ci.sh | 42 +-- scripts/ci/in_container/entrypoint_exec.sh | 2 +- scripts/ci/in_container/run_ci_tests.sh | 9 +- scripts/ci/libraries/_build_images.sh | 90 +++--- scripts/ci/libraries/_initialization.sh | 12 +- scripts/ci/libraries/_local_mounts.sh | 2 +- scripts/ci/libraries/_md5sum.sh | 4 +- scripts/ci/libraries/_push_pull_remove_images.sh | 5 +- scripts/ci/libraries/_start_end.sh | 4 + scripts/ci/libraries/_verbosity.sh | 11 + scripts/docker/entrypoint.sh | 110 ------- .../entrypoint_exec.sh => prod/clean-logs.sh} | 22 +- entrypoint.sh => scripts/prod/entrypoint_prod.sh | 0 26 files changed, 534 insertions(+), 651 deletions(-) diff --git a/.dockerignore b/.dockerignore index e7d6564..0a89434 100644 --- a/.dockerignore +++ b/.dockerignore @@ -49,7 +49,7 @@ !NOTICE !.github !requirements -!entrypoint.sh +!empty # Avoid triggering context change on README change (new companies using Airflow) # So please do not uncomment this line ;) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f2c96d1..ee799dd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,7 +33,7 @@ env: VERBOSE: "true" UPGRADE_TO_LATEST_REQUIREMENTS: "false" PYTHON_MAJOR_MINOR_VERSION: 3.5 - ENABLE_REGISTRY_CACHE: "true" + USE_GITHUB_REGISTRY: "true" CACHE_IMAGE_PREFIX: ${{ github.repository }} CACHE_REGISTRY_USERNAME: ${{ github.actor }} CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} diff --git a/BREEZE.rst b/BREEZE.rst index 0393369..3e9305a 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -293,7 +293,7 @@ Manage environments - CI (default) or Production - if ``--production-image`` fla Interact with CI environment: - * Run test target specified with ``breeze test-target`` command + * Run test target specified with ``breeze tests`` command * Execute arbitrary command in the test environment with ``breeze execute-command`` command * Execute arbitrary docker-compose command with ``breeze docker-compose`` command @@ -709,7 +709,6 @@ This is the current syntax for `./breeze <./breeze>`_: generate-requirements Generates pinned requirements for pip dependencies push-image Pushes images to registry initialize-local-virtualenv Initializes local virtualenv - kind-cluster Manages KinD cluster on the host setup-autocomplete Sets up autocomplete for breeze stop Stops the docker-compose environment restart Stops the docker-compose environment including DB cleanup @@ -719,11 +718,9 @@ This is the current syntax for `./breeze <./breeze>`_: Commands with arguments: docker-compose <ARG> Executes specified docker-compose command - execute-command <ARG> Executes specified command in the container kind-cluster <ARG> Manages KinD cluster on the host static-check <ARG> Performs selected static check for changed files - static-check-all-files <ARG> Performs selected static check for all files - test-target <ARG> Runs selected test target in the container + tests <ARG> Runs selected tests in the container Help commands: @@ -741,7 +738,7 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: shell - breeze shell [FLAGS] -- <EXTRA_ARGS> + breeze shell [FLAGS] [-- <EXTRA_ARGS>] This is default subcommand if no subcommand is used. @@ -758,6 +755,11 @@ This is the current syntax for `./breeze <./breeze>`_: and webserver ports are forwarded to appropriate database/webserver so that you can connect to it from your host environment. + You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is + especially useful to pass bash options, for example -c to execute command: + + 'breeze shell -- -c "ls -la"' + Flags: Run 'breeze flags' to see all applicable flags. @@ -795,6 +797,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: 2.7 3.5 3.6 3.7 @@ -858,10 +861,10 @@ This is the current syntax for `./breeze <./breeze>`_: -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: airflow. - -c, --registry-cache - If registry cache is enabled, pulls and pushes are done from the registry cache in github. - You need to be logged in to the registry in order to be able to pull/push from it and you - need to be committer to push to airflow registry. + -c, --github-registry + If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not + DockerHub. You need to be logged in to the registry in order to be able to pull/push from it + and you need to be committer to push to Apache Airflow' GitHub registry. -G, --github-organisation GitHub organisation used to pull, push images when cache is used. Default: apache. @@ -891,6 +894,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: 2.7 3.5 3.6 3.7 @@ -910,7 +914,7 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: exec - breeze exec + breeze exec [-- <EXTRA_ARGS>] Execs into interactive shell to an already running container. The container mus be started already by breeze shell command. If you are not familiar with tmux, this is the best @@ -936,6 +940,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: 2.7 3.5 3.6 3.7 @@ -955,7 +960,7 @@ This is the current syntax for `./breeze <./breeze>`_: breeze push_image [FLAGS] Pushes images to docker registry. You can push the images to DockerHub registry (default) - or to the GitHub cache registry (if --registry-cache flag is used). + or to the GitHub registry (if --github-registry flag is used). For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify the repository to push to. For GitHub repository --github-organisation and --github-repo @@ -968,8 +973,8 @@ This is the current syntax for `./breeze <./breeze>`_: 'breeze push-image' or 'breeze push-image --dockerhub-user user' to push to your private registry or 'breeze push-image --production-image' - to push production image or - 'breeze push-image --registry-cache' - to push to GitHub cache or - 'breeze push-image --registry-cache --github-organisation org' - for other organisation + 'breeze push-image --github-registry' - to push to GitHub image registry or + 'breeze push-image --github-registry --github-organisation org' - for other organisation Flags: @@ -979,10 +984,10 @@ This is the current syntax for `./breeze <./breeze>`_: -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: airflow. - -c, --registry-cache - If registry cache is enabled, pulls and pushes are done from the registry cache in github. - You need to be logged in to the registry in order to be able to pull/push from it and you - need to be committer to push to airflow registry. + -c, --github-registry + If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not + DockerHub. You need to be logged in to the registry in order to be able to pull/push from it + and you need to be committer to push to Apache Airflow' GitHub registry. -G, --github-organisation GitHub organisation used to pull, push images when cache is used. Default: apache. @@ -1014,76 +1019,11 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. - One of: - - 2.7 3.5 3.6 3.7 - - #################################################################################################### - - - Detailed usage for command: kind-cluster - - - breeze kind-cluster [FLAGS] OPERATION - - Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests. - It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it. - This enables you to run tests inside the breeze environment with latest airflow images loaded. - Note that in case of deploying airflow, the first step is to rebuild the image and loading it - to the cluster so you can also pass appropriate build image flags that will influence - rebuilding the production image. Operation is one of: - - start stop restart status deploy test - - Flags: - - -p, --python <PYTHON_MAJOR_MINOR_VERSION> - Python version used for the image. This is always major/minor version. One of: 2.7 3.5 3.6 3.7 - -F, --force-build-images - Forces building of the local docker images. The images are rebuilt - automatically for the first time or when changes are detected in - package-related files, but you can force it using this flag. - - -P, --force-pull-images - Forces pulling of images from DockerHub before building to populate cache. The - images are pulled by default only for the first time you run the - environment, later the locally build images are used as cache. - - -E, --extras - Extras to pass to build images The default are different for CI and production images: - - CI image: - devel_ci - - Production image: - async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack, - ssh,statsd,virtualenv - - --additional-extras - Additional extras to pass to build images The default is no additional extras. - - --additional-python-deps - Additional python dependencies to use when building the images. - - --additional-dev-deps - Additional apt dev dependencies to use when building the images. - - --additional-runtime-deps - Additional apt runtime dependencies to use when building the images. - - -C, --force-clean-images - Force build images with cache disabled. This will remove the pulled or build images - and start building images from scratch. This might take a long time. - - -L, --use-local-cache - Uses local cache to build images. No pulled images will be used, but results of local - builds in the Docker cache are used instead. - #################################################################################################### @@ -1156,7 +1096,7 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: docker-compose - breeze docker-compose [FLAGS] COMMAND -- <EXTRA_ARGS> + breeze docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>] Run docker-compose command instead of entering the environment. Use 'help' as command to see available commands. The <EXTRA_ARGS> passed after -- are treated @@ -1168,54 +1108,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. - One of: - 2.7 3.5 3.6 3.7 - - -b, --backend <BACKEND> - Backend to use for tests - it determines which database is used. - One of: - - sqlite mysql postgres - - Default: sqlite - - --postgres-version <POSTGRES_VERSION> - Postgres version used. One of: - - 9.6 10 - - --mysql-version <MYSQL_VERSION> - Mysql version used. One of: - - 5.6 5.7 - - -v, --verbose - Show verbose information about executed commands (enabled by default for running test). - Note that you can further increase verbosity and see all the commands executed by breeze - by running 'export VERBOSE_COMMANDS="true"' before running breeze. - - - #################################################################################################### - - - Detailed usage for command: execute-command - - - breeze execute-command [FLAGS] COMMAND -- <EXTRA_ARGS> - - Run chosen command instead of entering the environment. The command is run using - 'bash -c "<command with args>" if you need to pass arguments to your command, you need - to pass them together with command surrounded with " or '. Alternatively you can - pass arguments as <EXTRA_ARGS> passed after --. For example: - - 'breeze execute-command "ls -la"' or - 'breeze execute-command ls -- --la' - - Flags: - - -p, --python <PYTHON_MAJOR_MINOR_VERSION> - Python version used for the image. This is always major/minor version. One of: 2.7 3.5 3.6 3.7 @@ -1265,6 +1158,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: 2.7 3.5 3.6 3.7 @@ -1316,7 +1210,7 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: static-check - breeze static-check [FLAGS] STATIC_CHECK + breeze static-check [FLAGS] STATIC_CHECK [-- <EXTRA_ARGS>] Run selected static checks for currently changed files. You should specify static check that you would like to run or 'all' to run all checks. One of: @@ -1334,6 +1228,7 @@ This is the current syntax for `./breeze <./breeze>`_: 'breeze static-check mypy' or 'breeze static-check mypy -- --files tests/core.py' + 'breeze static-check mypy -- --all-files' You can see all the options by adding --help EXTRA_ARG: @@ -1343,46 +1238,18 @@ This is the current syntax for `./breeze <./breeze>`_: #################################################################################################### - Detailed usage for command: static-check-all-files - - - breeze static-check-all [FLAGS] STATIC_CHECK - - Run selected static checks for all applicable files. You should specify static check that - you would like to run or 'all' to run all checks. One of: - - all airflow-config-yaml bat-tests build check-apache-license - check-executables-have-shebangs check-hooks-apply check-integrations - check-merge-conflict check-xml debug-statements detect-private-key doctoc - end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs insert-license - language-matters lint-dockerfile mixed-line-ending mypy pydevd python2-compile - python2-fastcheck python-no-log-warn rst-backticks setup-order shellcheck - trailing-whitespace update-breeze-file update-extras update-local-yml-file yamllint - - You can pass extra arguments including options to the pre-commit framework as - <EXTRA_ARGS> passed after --. For example: - - 'breeze static-check-all-files mypy' or - 'breeze static-check-all-files mypy -- --verbose' - - You can see all the options by adding --help EXTRA_ARG: - - 'breeze static-check-all-files mypy -- --help' - - - #################################################################################################### - - - Detailed usage for command: test-target + Detailed usage for command: tests - breeze test-target [FLAGS] TEST_TARGET -- <EXTRA_ARGS> + breeze tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>] Run the specified unit test target. There might be multiple targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated - as additional options passed to pytest. For example: + as additional options passed to pytest. You can pass 'tests' as target to + run all tests. For example: - 'breeze test-target tests/test_core.py -- --logging-level=DEBUG' + 'breeze tests tests/test_core.py -- --logging-level=DEBUG' + 'breeze tests tests Flags: @@ -1432,6 +1299,7 @@ This is the current syntax for `./breeze <./breeze>`_: -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: 2.7 3.5 3.6 3.7 @@ -1594,10 +1462,10 @@ This is the current syntax for `./breeze <./breeze>`_: -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: airflow. - -c, --registry-cache - If registry cache is enabled, pulls and pushes are done from the registry cache in github. - You need to be logged in to the registry in order to be able to pull/push from it and you - need to be committer to push to airflow registry. + -c, --github-registry + If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not + DockerHub. You need to be logged in to the registry in order to be able to pull/push from it + and you need to be committer to push to Apache Airflow' GitHub registry. -G, --github-organisation GitHub organisation used to pull, push images when cache is used. Default: apache. @@ -1621,17 +1489,6 @@ This is the current syntax for `./breeze <./breeze>`_: .. END BREEZE HELP MARKER -Convenience Scripts -------------------- - -Once you run ``./breeze`` you can also execute various actions via generated convenience scripts: - -.. code-block:: - - Enter the environment : ./.build/cmd_run - Run command in the environment : ./.build/cmd_run "[command with args]" [bash options] - Run tests in the environment : ./.build/test_run [test-target] [pytest options] - Run Docker compose command : ./.build/dc [help/pull/...] [docker-compose options] Troubleshooting =============== diff --git a/Dockerfile b/Dockerfile index 7c722cf..56fdf0f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -158,6 +158,23 @@ ENV PIP_VERSION=${PIP_VERSION} RUN pip install --upgrade pip==${PIP_VERSION} +ARG AIRFLOW_REPO=apache/airflow +ENV AIRFLOW_REPO=${AIRFLOW_REPO} + +ARG AIRFLOW_BRANCH=master +ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH} + +ARG AIRFLOW_EXTRAS +ARG ADDITIONAL_AIRFLOW_EXTRAS="" +ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} + +# In case of Production build image segment we want to pre-install master version of airflow +# dependencies from github so that we do not have to always reinstall it from the scratch. +RUN pip install --user \ + "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]"; \ + --constraint "https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"; \ + && pip uninstall --yes apache-airflow; + ARG AIRFLOW_SOURCES_FROM="." ENV AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM} @@ -172,10 +189,6 @@ ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY} ARG AIRFLOW_VERSION ENV AIRFLOW_VERSION=${AIRFLOW_VERSION} -ARG AIRFLOW_EXTRAS -ARG ADDITIONAL_AIRFLOW_EXTRAS="" -ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} - ARG ADDITIONAL_PYTHON_DEPS="" ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS} @@ -215,13 +228,6 @@ RUN \ rm -rf "${WWW_DIR}/node_modules"; \ fi -ARG ENTRYPOINT_FILE="entrypoint.sh" -ENV ENTRYPOINT_FILE="${ENTRYPOINT_FILE}" - -# hadolint ignore=DL3020 -ADD ${ENTRYPOINT_FILE} /entrypoint -RUN chmod a+x /entrypoint - ############################################################################################## # This is the actual Airflow image - much smaller than the build one. We copy # installed Airflow and all it's dependencies from the build image to make it smaller. @@ -334,7 +340,11 @@ RUN mkdir -pv "${AIRFLOW_HOME}"; \ chown -R "airflow" "${AIRFLOW_HOME}" COPY --chown=airflow:airflow --from=airflow-build-image /root/.local "/home/airflow/.local" -COPY --chown=airflow:airflow --from=airflow-build-image /entrypoint /entrypoint + +COPY scripts/prod/entrypoint_prod.sh /entrypoint +COPY scripts/prod/clean-logs.sh /clean-logs + +RUN chmod a+x /entrypoint /clean-logs USER airflow diff --git a/Dockerfile.ci b/Dockerfile.ci index 4c9741b..232711b 100644 --- a/Dockerfile.ci +++ b/Dockerfile.ci @@ -222,10 +222,11 @@ ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH} # In case of CI builds we want to pre-install master version of airflow dependencies so that # We do not have to always reinstall it from the scratch. # This can be reinstalled from latest master by increasing PIP_DEPENDENCIES_EPOCH_NUMBER. -# And is automatically reinstalled from the scratch with every python patch level release -RUN pip install "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]"; \ +# And is automatically reinstalled from the scratch every time patch release of python gets released +RUN pip install \ + "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]"; \ --constraint "https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"; \ - && pip uninstall --yes apache-airflow + && pip uninstall --yes apache-airflow; # Link dumb-init for backwards compatibility (so that older images also work) RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init @@ -274,7 +275,8 @@ COPY airflow/www_rbac/static ${AIRFLOW_SOURCES}/airflow/www_rbac/static/ # Package JS/css for production RUN yarn --cwd airflow/www_rbac run prod -COPY entrypoint.sh /entrypoint.sh +COPY scripts/ci/in_container/entrypoint_ci.sh /entrypoint +RUN chmod a+x /entrypoint # Copy selected subdirectories only COPY .github/ ${AIRFLOW_SOURCES}/.github/ @@ -290,9 +292,6 @@ COPY .coveragerc .rat-excludes .flake8 LICENSE MANIFEST.in NOTICE CHANGELOG.txt setup.cfg setup.py \ ${AIRFLOW_SOURCES}/ -# Needed for building images via docker-in-docker inside the docker -COPY Dockerfile.ci ${AIRFLOW_SOURCES}/Dockerfile.ci - # Install autocomplete for airflow RUN register-python-argcomplete airflow >> ~/.bashrc @@ -316,6 +315,4 @@ ENV PATH="${HOME}:${PATH}" EXPOSE 8080 -ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint.sh"] - -CMD ["--help"] +ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"] diff --git a/IMAGES.rst b/IMAGES.rst index 3add528..4cdf86d 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -53,13 +53,13 @@ also change the repository itself by adding ``--dockerhub-user`` and ``--dockerh You can build the CI image using this command: -.. code-block:: +.. code-block:: bash ./breeze build-image You can build production image using this command: -.. code-block:: +.. code-block:: bash ./breeze build-image --production-image @@ -73,7 +73,7 @@ can change the extras via the ``--extras`` parameters. You can see default extra For example if you want to build python 3.7 version of production image with "all" extras installed you should run this command: -.. code-block:: +.. code-block:: bash ./breeze build-image --python 3.7 --extras "all" --production-image @@ -90,42 +90,132 @@ In Breeze by default, the airflow is installed using local sources of Apache Air You can also build production images from PIP packages via providing ``--install-airflow-version`` parameter to Breeze: -.. code-block:: +.. code-block:: bash ./breeze build-image --python 3.7 --extras=gcp --production-image --install-airflow-version=1.10.9 This will build the image using command similar to: -.. code-block:: +.. code-block:: bash - pip install apache-airflow[gcp]==1.10.9 \ + pip install apache-airflow[sendgrid]==1.10.9 \ --constraint https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python3.7.txt -This will also download entrypoint script from https://raw.githubusercontent.com/apache/airflow/v1-10-test/entrypoint.sh -url. It is important so that we have matching version of the requirements. - -The requirement files and entrypoint only appeared in version 1.10.10 of airflow so if you install +The requirement files only appeared in version 1.10.10 of airflow so if you install an earlier version - both constraint and requirements should point to 1.10.10 version. You can also build production images from specific Git version via providing ``--install-airflow-reference`` parameter to Breeze: -.. code-block:: +.. code-block:: bash pip install https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow \ --constraint https://raw.githubusercontent.com/apache/airflow/<tag>/requirements/requirements-python3.7.txt -This will also Download entrypoint script from ``https://raw.githubusercontent.com/apache/airflow/<tag>/entrypoint.sh`` -url. +Using cache during builds +========================= + +Default mechanism used in Breeze for building images uses - as base - images puled from DockerHub or +GitHub Image Registry. This is in order to speed up local builds and CI builds - instead of 15 minutes +for rebuild of CI images, it takes usually less than 3 minutes when cache is used. For CI builds this is +usually the best strategy - to use default "pull" cache - same for Production Image - it's better to rely +on the "pull" mechanism rather than rebuild the image from the scratch. + +However when you are iterating on the images and want to rebuild them quickly and often you can provide the +``--use-local-cache`` flag to build commands - this way the standard docker mechanism based on local cache +will be used. The first time you run it, it will take considerably longer time than if you use the +default pull mechanism, but then when you do small, incremental changes to local sources, Dockerfile image +and scripts further rebuilds with --use-local-cache will be considerably faster. + +.. code-block:: bash + + ./breeze build-image --python 3.7 --production-image --use-local-cache + +You can also turn local docker caching by setting DOCKER_CACHE variable to "local" instead of the default +"pulled" and export it to Breeze. + +.. code-block:: bash + + export DOCKER_CACHE="local" + +You can also - if you really want - disable caching altogether by setting this variable to "no-cache". +This is how "scheduled" builds in our CI are run - those builds take a long time because they +always rebuild everything from scratch. + +.. code-block:: bash + + export DOCKER_CACHE="no-cache" + + +Choosing image registry +======================= + +By default images are pulled and pushed from and to DockerHub registry when you use Breeze's push-image +or build commands. + +Our images are named like that: + +.. code-block:: bash + + apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y - for production images + apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y-ci - for CI images + apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y-build - for production build stage + +For example: + +.. code-block:: bash + + apache/airflow:master-python3.6 - production "latest" image from current master + apache/airflow:master-python3.6-ci - CI "latest" image from current master + apache/airflow:v1-10-test-python2.7-ci - CI "latest" image from current v1-10-test branch + apache/airflow:1.10.10-python3.6 - production image for 1.10.10 release + apache/airflow:1.10.10-1-python3.6 - production image for 1.10.10 with some patches applied + + +You can see DockerHub images at `<https://hub.docker.com/repository/docker/apache/airflow>`_ + +By default DockerHub registry is used when you push or pull such images. +However for CI builds we keep the images in GitHub registry as well - this way we can easily push +the images automatically after merge requests and use such images for Pull Requests +as cache - which makes it much it much faster for CI builds (images are available in cache +right after merged request in master finishes it's build), The difference is visible especially if +significant changes are done in the Dockerfile.CI. + +The images are named differently (in Docker definition of image names - registry URL is part of the +image name if DockerHub is not used as registry). Also GitHub has its own structure for registries +each project has its own registry naming convention that should be followed. The name of +images for GitHub registry are: + +.. code-block:: bash + + docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y - for production images + docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-ci - for CI images + docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-build - for production build state + +Note that we never push or pull TAG images to GitHub registry. It is only used for CI builds + +You can see all the current GitHub images at `<https://github.com/apache/airflow/packages>`_ + +In order to interact with the GitHub images you need to add ``--github-registry`` flag to the pull/push +commands in Breeze. This way the images will be pulled/pushed from/to GitHub rather than from/to +DockerHub. Images are build locally as ``apache/airflow`` images but then they are tagged with the right +GitHub tags for you. + +You can read more about the CI configuration and how CI builds are using DockerHub/GitHub images +in `<CI.rst>`_. + +Note that you need to be committer and have the right to push to DockerHub and GitHub and you need to +be logged in. Only committers can push images directly. + Technical details of Airflow images =================================== The CI image is used by Breeze as shell image but it is also used during CI build. The image is single segment image that contains Airflow installation with "all" dependencies installed. -It is optimised for rebuild speed It installs PIP dependencies from the current branch first - -so that any changes in setup.py do not trigger -reinstalling of all dependencies. There is a second step of installation that re-installs the dependencies +It is optimised for rebuild speed. It installs PIP dependencies from the current branch first - +so that any changes in setup.py do not trigger reinstalling of all dependencies. +There is a second step of installation that re-installs the dependencies from the latest sources so that we are sure that latest dependencies are installed. The production image is a multi-segment image. The first segment "airflow-build-image" contains all the @@ -135,6 +225,11 @@ build it from local sources. This is particularly useful in CI environment where to run Kubernetes tests. See below for the list of arguments that should be provided to build production image from the local sources. +The image is primarily optimised for size of the final image, but also for speed of rebuilds - the +'airlfow-build-image' segment uses the same technique as the CI builds for pre-installing PIP dependencies. +It first pre-installs them from the right github branch and only after that final airflow installation is +done from either local sources or remote location (PIP or github repository). + Manually building the images ---------------------------- @@ -180,11 +275,10 @@ The following build arguments (``--build-arg`` in docker build command) can be u | | | done for cassandra driver (much faster) | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_REPO`` | ``apache/airflow`` | the repository from which PIP | -| | | dependencies are installed (CI | -| | | optimised) | +| | | dependencies are pre-installed | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_BRANCH`` | ``master`` | the branch from which PIP dependencies | -| | | are installed (CI optimised) | +| | | are pre-installed | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_CI_BUILD_EPOCH`` | ``1`` | increasing this value will reinstall PIP | | | | dependencies from the repository from | @@ -192,9 +286,9 @@ The following build arguments (``--build-arg`` in docker build command) can be u +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_EXTRAS`` | ``all`` | extras to install | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_AIRFLOW_EXTRAS`` | ```` | additional extras to install | +| ``ADDITIONAL_AIRFLOW_EXTRAS`` | | additional extras to install | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_PYTHON_DEPS`` | \```\` | additional python dependencies to | +| ``ADDITIONAL_PYTHON_DEPS`` | | additional python dependencies to | | | | install | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to | @@ -208,7 +302,7 @@ Here are some examples of how CI images can built manually. CI is always built f This builds the CI image in version 3.7 with default extras ("all"). -.. code-block:: +.. code-block:: bash docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 @@ -216,7 +310,7 @@ This builds the CI image in version 3.7 with default extras ("all"). This builds the CI image in version 3.6 with "gcp" extra only. -.. code-block:: +.. code-block:: bash docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg AIRFLOW_EXTRAS=gcp @@ -224,14 +318,14 @@ This builds the CI image in version 3.6 with "gcp" extra only. This builds the CI image in version 3.6 with "apache-beam" extra added. -.. code-block:: +.. code-block:: bash docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" This builds the CI image in version 3.6 with "mssql" additional package added. -.. code-block:: +.. code-block:: bash docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_PYTHON_DEPS="mssql" @@ -270,8 +364,11 @@ The following build arguments (``--build-arg`` in docker build command) can be u | ``AIRFLOW_ORG`` | ``apache`` | Github organisation from which Airflow | | | | is installed (when installed from repo) | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_REPO`` | ``airflow`` | Github repository from which Airflow is | -| | | installed (when installed from repo) | +| ``AIRFLOW_REPO`` | ``apache/airflow`` | the repository from which PIP | +| | | dependencies are pre-installed | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_BRANCH`` | ``master`` | the branch from which PIP dependencies | +| | | are pre-installed | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_GIT_REFERENCE`` | ``master`` | reference (branch or tag) from Github | | | | repository from which Airflow is | @@ -285,10 +382,10 @@ The following build arguments (``--build-arg`` in docker build command) can be u | ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is | | | | installed | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_AIRFLOW_EXTRAS`` | ```` | Optional additional extras with which | +| ``ADDITIONAL_AIRFLOW_EXTRAS`` | | Optional additional extras with which | | | | airflow is installed | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_PYTHON_DEPS`` | ```` | Optional python packages to extend | +| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend | | | | the image with some extra dependencies | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to | @@ -342,25 +439,20 @@ production image. There are three types of build: | | the package or from GitHub URL. | | | See examples below | +-----------------------------------+-----------------------------------+ -| ``ENTRYPOINT_FILE`` | Should point to entrypoint.sh | -| | file in case of installation from | -| | the package or from GitHub URL. | -| | See examples below | -+-----------------------------------+-----------------------------------+ | ``AIRFLOW_WWW`` | In case of Airflow 2.0 it should | | | be "www", in case of Airflow 1.10 | | | series it should be "www_rbac". | | | See examples below | +-----------------------------------+-----------------------------------+ | ``AIRFLOW_SOURCES_FROM`` | Sources of Airflow. Set it to | -| | "entrypoint.sh" to avoid costly | +| | "empty" to avoid costly | | | Docker context copying | | | in case of installation from | | | the package or from GitHub URL. | | | See examples below | +-----------------------------------+-----------------------------------+ | ``AIRFLOW_SOURCES_TO`` | Target for Airflow sources. Set | -| | to "/entrypoint" to avoid costly | +| | to "/empty" to avoid costly | | | Docker context copying | | | in case of installation from | | | the package or from GitHub URL. | @@ -368,9 +460,10 @@ production image. There are three types of build: +-----------------------------------+-----------------------------------+ -This builds production image in version 3.6 with default extras from the local sources: +This builds production image in version 3.6 with default extras from the local sources (master version +of 2.0 currently): -.. code-block:: +.. code-block:: bash docker build . @@ -379,46 +472,48 @@ requirements taken from v1-10-test branch in Github. Note that versions 1.10.9 and below have no requirements so requirements should be taken from head of the 1.10.10 tag. -.. code-block:: +.. code-block:: bash docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.10.tar.gz#egg=apache-airflow"; \ --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"; \ - --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"; \ - --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \ - --build-arg AIRFLOW_SOURCES_TO="/entrypoint" + --build-arg AIRFLOW_BRANCH="v1-10-test" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" This builds the production image in version 3.7 with default extras from 1.10.10 Pypi package and -requirements taken from v1-10-test branch in Github. +requirements taken from 1.10.10 tag in Github and pre-installed pip dependencies from the top +of v1-10-test branch. -.. code-block:: +.. code-block:: bash docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \ + --build-arg AIRFLOW_BRANCH="v1-10-test" \ --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"; \ - --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"; \ - --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \ - --build-arg AIRFLOW_SOURCES_TO="/entrypoint" + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and -additional python dependencies. +additional python dependencies and pre-installed pip dependencies from the top +of v1-10-test branch. -.. code-block:: +.. code-block:: bash docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \ + --build-arg AIRFLOW_BRANCH="v1-10-test" \ --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"; \ - --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"; \ - --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \ - --build-arg AIRFLOW_SOURCES_TO="/entrypoint" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client" @@ -446,7 +541,7 @@ Image manifests Together with the main CI images we also build and push image manifests. Those manifests are very small images that contain only results of the docker inspect for the image. This is in order to be able to determine very quickly if the image in the docker registry has changed a lot since the last time. -Unfortunately docker registry (specifically dockerhub registry) has no anonymous way of querying image +Unfortunately docker registry (specifically DockerHub registry) has no anonymous way of querying image details via API, you need to download the image to inspect it. We overcame it in the way that always when we build the image we build a very small image manifest and push it to registry together with the main CI image. The tag for the manifest image is the same as for the image it refers @@ -465,29 +560,158 @@ You can do it via the ``--force-pull-images`` flag to force pulling the latest i For production image: -.. code-block:: +.. code-block:: bash ./breeze build-image --force-pull-images --production-image For CI image Breeze automatically uses force pulling in case it determines that your image is very outdated, however uou can also force it with the same flag. -.. code-block:: +.. code-block:: bash ./breeze build-image --force-pull-images -Using the images -================ -Both images have entrypoint set as dumb-init with entrypoint.sh script executed (in order to forward -signals). This entrypoint works as follows: +Embedded image scripts +====================== + +Both images have a set of scripts that can be used in the image. Those are: + * /entrypoint - entrypoint script used when entering the image + * /clean-logs - script for periodic log cleaning + + +Running the CI image +==================== + +The entrypoint in the CI image contains all the initialisation needed for tests to be immediately executed. +It is copied from ``scripts/ci/in_container/entrypoint_ci.sh``. + +The default behaviour is that you are dropped into bash shell. However if RUN_TESTS variable is +set to "true", then tests passed as arguments are executed + +The entrypoint performs those operations: + +* checks if the environment is ready to test (including database and all integrations). It waits + until all the components are ready to work + +* installs older version of Airflow (if older version of Airflow is requested to be installed + via ``INSTALL_AIRFLOW_VERSION`` variable. + +* Sets up Kerberos if Kerberos integration is enabled (generates and configures Kerberos token) + +* Sets up ssh keys for ssh tests and restarts teh SSH server + +* Sets all variables and configurations needed for unit tests to run + +* Reads additional variables set in ``files/airflow-breeze-config/variables.env`` by sourcing that file + +* In case of CI run sets parallelism to 2 to avoid excessive number of processes to run + +* In case of CI run sets default parameters for pytest + +* In case of running integration/long_running/quarantined tests - it sets the right pytest flags + +* Sets default "tests" target in case the target is not explicitly set as additional argument + +* Runs system tests if RUN_SYSTEM_TESTS flag is specified, otherwise runs regular unit and integration tests + + +Using the PROD image +==================== + +The PROD image entrypoint works as follows: * If ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is passed to the container and it is either mysql or postgres SQL alchemy connection, then the connection is checked and the script waits until the database is reachable. + * If no ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is set or if it is set to sqlite SQL alchemy connection then db reset is executed. + * If ``AIRFLOW__CELERY__BROKER_URL`` variable is passed and scheduler, worker of flower command is used then the connection is checked and the script waits until the Celery broker database is reachable. -* If no argument is specified - you are dropped in bash shell. -* If there are any arguments they are passed to "airflow" command +* If first argument equals to "bash" - it dropped in bash shell or executes bash command if you specify + extra arguments. For example: + +.. code-block:: bash + + docker run -it apache/airflow:master-python3.6 bash -c "ls -la" + total 16 + drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . + drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 dags + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 logs + +* If first argument is equal to "python" - you are dropped in python shell or python commands are executed if + you pass extra parameters. For example: + +.. code-block:: bash + + > docker run -it apache/airflow:master-python3.6 python -c "print('test')" + test + +* If there are any other arguments - they are passed to "airflow" command + +.. code-block:: bash + + > docker run -it apache/airflow:master-python3.6 --help + + usage: airflow [-h] + {celery,config,connections,dags,db,info,kerberos,plugins,pools,roles,rotate_fernet_key,scheduler,sync_perm,tasks,users,variables,version,webserver} + ... + + positional arguments: + + Groups: + celery Start celery components + connections List/Add/Delete connections + dags List and manage DAGs + db Database operations + pools CRUD operations on pools + roles Create/List roles + tasks List and manage tasks + users CRUD operations on users + variables CRUD operations on variables + + Commands: + config Show current application configuration + info Show information about current Airflow and environment + kerberos Start a kerberos ticket renewer + plugins Dump information about loaded plugins + rotate_fernet_key Rotate encrypted connection credentials and variables + scheduler Start a scheduler instance + sync_perm Update permissions for existing roles and DAGs + version Show the version + webserver Start a Airflow webserver instance + + optional arguments: + -h, --help show this help message and exit + + +Alpha versions of 1.10.10 production-ready images +================================================= + +The production images have been released for the first time in 1.10.10 release of Airflow as "Alpha" quality +ones. Between 1.10.10 the images are being improved and the 1.10.10 images should be patched and +published several times separately in order to test them with the upcoming Helm Chart. + +Those images are for development and testing only and should not be used outside of the +development community. + +The images were pushed with tags following the pattern: ``apache/airflow:1.10.10.1-alphaN-pythonX.Y``. +Patch level is an increasing number (starting from 1). + +Those are alpha-quality releases however they contain the officially released Airflow ``1.10.10`` code. +The main changes in the images are scripts embedded in the images. + +The following versions were pushed: + ++-------+--------------------------------+----------------------------------------------------------+ +| Patch | Tag pattern | Description | ++=======+================================+==========================================================+ +| 1 | ``1.10.10.1-alpha1-pythonX.Y`` | Support for parameters added to bash and python commands | ++-------+--------------------------------+----------------------------------------------------------+ +| 2 | ``1.10.10-1-alpha2-pythonX.Y`` | Added "/clean-logs" script | ++-------+--------------------------------+----------------------------------------------------------+ + +The commits used to generate those images are tagged with ``prod-image-1.10.10.1-alphaN`` tags. diff --git a/TESTING.rst b/TESTING.rst index ea6884e..36c1427 100644 --- a/TESTING.rst +++ b/TESTING.rst @@ -131,23 +131,23 @@ Running Tests for a Specified Target Using Breeze from the Host --------------------------------------------------------------- If you wish to only run tests and not to drop into shell, apply the -``-t``, ``--test-target`` flag. You can add extra pytest flags after ``--`` in the command line. +``tests`` command. You can add extra targets and pytest flags after the ``tests`` command. .. code-block:: bash - ./breeze test-target tests/hooks/test_druid_hook.py -- --logging-level=DEBUG + ./breeze tests tests/hooks/test_druid_hook.py tests/tests_core.py --logging-level=DEBUG -You can run the whole test suite with a special '.' test target: +You can run the whole test suite with a 'tests' test target: .. code-block:: bash - ./breeze test-target . + ./breeze tests tests You can also specify individual tests or a group of tests: .. code-block:: bash - ./breeze test-target tests/test_core.py::TestCore + ./breeze tests tests/test_core.py::TestCore Airflow Integration Tests diff --git a/breeze b/breeze index 9e1c746..d78eaf2 100755 --- a/breeze +++ b/breeze @@ -68,7 +68,7 @@ function setup_default_breeze_variables() { # We have different versions of images depending on the python version used. We keep up with the # Latest patch-level changes in Python (this is done automatically during CI builds) so we have # To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor - # version of python in X.Y format (3.6, 3.7 etc). + # version of python in X.Y format (2.7, 3.5, 3.6, 3.7 etc). export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}" # When we generate documentation for README files, we want to force the width of terminal so that @@ -103,12 +103,6 @@ function setup_default_breeze_variables() { # Determines if help should be run (set to true by --help flag) RUN_HELP="false" - # Holds chosen command to run in case 'execute-command' command is used. - RUN_COMMAND="" - - # Holds the test target if the 'test-target' command is used. - TEST_TARGET="" - # Holds docker compose command if the `docker-compose` command is used. DOCKER_COMPOSE_COMMAND="" @@ -426,12 +420,24 @@ function prepare_command_file() { local TESTS="${3}" local COMPOSE_FILE="${4}" local AIRFLOW_IMAGE="${5}" - local EXPANSION="${6-@}" cat <<EOF > "${FILE}" #!/usr/bin/env bash -cd "\$(pwd)" || exit +if [[ \${VERBOSE} == "true" ]]; then + echo + echo "Executing script:" + echo + echo "\${BASH_SOURCE[0]} \${@}" + echo + set -x +fi +cd "\$( dirname "\${BASH_SOURCE[0]}" )" || exit export DOCKERHUB_USER=${DOCKERHUB_USER} export DOCKERHUB_REPO=${DOCKERHUB_REPO} +HOST_USER_ID=\$(id -ur) +export HOST_USER_ID +HOST_GROUP_ID=\$(id -gr) +export HOST_GROUP_ID +export HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" export COMPOSE_FILE="${COMPOSE_FILE}" export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" export BACKEND="${BACKEND}" @@ -446,10 +452,10 @@ export MYSQL_VERSION="${MYSQL_VERSION}" export AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" export MYSQL_ENCODING="${MYSQL_ENCODING}" export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}" -export AIRFOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" +export AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" export AIRFLOW_IMAGE="${AIRFLOW_IMAGE}" export SQLITE_URL="${SQLITE_URL}" -docker-compose --log-level INFO ${CMD}\$${EXPANSION}" +docker-compose --log-level INFO ${CMD} EOF chmod u+x "${FILE}" } @@ -502,43 +508,22 @@ function prepare_command_files() { export COMPOSE_CI_FILE export COMPOSE_PROD_FILE - CI_ENTRYPOINT_FILE="/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh" - PROD_ENTRYPOINT_FILE="/entrypoint" - # Base python image for the build export PYTHON_BASE_IMAGE=python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci" export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}" export BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}" - DC_RUN_CI_COMMAND="run --service-ports --rm airflow \"${CI_ENTRYPOINT_FILE} " - DC_RUN_PROD_COMMAND="run --service-ports --rm airflow \"${PROD_ENTRYPOINT_FILE} " - - LAST_DC_RUN_CI_FILE="cmd_run_ci" - LAST_DC_RUN_PROD_FILE="cmd_run_prod" - LAST_DC_TEST_CI_FILE="test_run_ci" LAST_DC_CI_FILE="dc_ci" LAST_DC_PROD_FILE="dc_prod" - # Prepare script for "run ci command" - prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_RUN_CI_FILE}" \ - "${DC_RUN_CI_COMMAND}" "false" "${COMPOSE_CI_FILE}" "${AIRFLOW_CI_IMAGE}" '*' - - # Prepare script for "run prod command" - prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_RUN_PROD_FILE}" \ - "${DC_RUN_PROD_COMMAND}" "false" "${COMPOSE_PROD_FILE}" "${AIRFLOW_PROD_IMAGE}" '*' - - # Prepare script for "run test" - prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_TEST_CI_FILE}" \ - "${DC_RUN_CI_COMMAND}" "true" "${COMPOSE_CI_FILE}" "${AIRFLOW_CI_IMAGE}" '*' - - # Prepare script for "run docker compose command" + # Prepare script for "run docker compose CI command" prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE}" \ - '"' "false" "${COMPOSE_CI_FILE}" "${AIRFLOW_CI_IMAGE}" + "\"\${@}\"" "false" "${COMPOSE_CI_FILE}" "${AIRFLOW_CI_IMAGE}" - # Prepare script for "run docker compose prod command" + # Prepare script for "run docker compose PROD command" prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_PROD_FILE}" \ - '"' "false" "${COMPOSE_PROD_FILE}" "${AIRFLOW_PROD_IMAGE}" + "\"\${@}\"" "false" "${COMPOSE_PROD_FILE}" "${AIRFLOW_PROD_IMAGE}" } # Prints detailed help for all commands and flgas. Used to generate documentation added to BREEZE.rst @@ -747,11 +732,11 @@ function parse_arguments() { echo export FORWARD_CREDENTIALS="true" shift 1 ;; - -c|--registry-cache) + -c|--github-registry) echo - echo "Use cache for the container registry" + echo "Use github registry" echo - export ENABLE_REGISTRY_CACHE="true" + export USE_GITHUB_REGISTRY="true" shift ;; -G|--github-organisation) echo @@ -819,10 +804,6 @@ function parse_arguments() { fi COMMAND_TO_RUN="run_docker_compose" ;; - execute-command) - LAST_SUBCOMMAND="${1}" - COMMAND_TO_RUN="run_in_bash" - shift ;; generate-requirements) LAST_SUBCOMMAND="${1}" COMMAND_TO_RUN="perform_generate_requirements" @@ -834,7 +815,7 @@ function parse_arguments() { push-image) LAST_SUBCOMMAND="${1}" COMMAND_TO_RUN="perform_push_image" - SKIP_CHECK_REMOTE_IMAGE="true" + export SKIP_CHECK_REMOTE_IMAGE="true" shift ;; initialize-local-virtualenv) LAST_SUBCOMMAND="${1}" @@ -885,33 +866,6 @@ function parse_arguments() { shift 2 fi ;; - static-check-all-files) - LAST_SUBCOMMAND="${1}" - COMMAND_TO_RUN="perform_static_checks" - if [[ "$#" -lt 2 ]]; then - if [[ ${RUN_HELP} != "true" ]]; then - echo "You should specify static check that you would like to run or 'all' to run all checks." - echo - echo "One of :" - echo - echo "${_BREEZE_ALLOWED_STATIC_CHECKS:=}" - echo - echo "For example:" - echo - echo "${CMDNAME} static-check-all-files mypy" - echo - exit 1 - else - shift - fi - else - export PYTHON_MAJOR_MINOR_VERSION=${STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION} - export STATIC_CHECK="${2:-}" - export STATIC_CHECK_ALL_FILES="true" - EXTRA_STATIC_CHECK_OPTIONS+=("--all-files" "--show-diff-on-failure") - shift 2 - fi - ;; stop) LAST_SUBCOMMAND="${1}" COMMAND_TO_RUN="run_docker_compose" @@ -926,14 +880,11 @@ function parse_arguments() { SECOND_COMMAND_TO_RUN="enter_breeze" echo "Restarts the environment. Includes emptying the databases." shift ;; - test-target) + tests) LAST_SUBCOMMAND="${1}" if [[ $# -lt 2 ]]; then RUN_HELP="true" shift - else - export TEST_TARGET="${2}" - shift 2 fi COMMAND_TO_RUN="run_tests" ;; toggle-suppress-cheatsheet) @@ -1033,7 +984,6 @@ function prepare_usage() { export USAGE_BUILD_IMAGE="Builds CI or Production docker image" export USAGE_CLEANUP_IMAGE="Cleans up the container image created" export USAGE_DOCKER_COMPOSE="Executes specified docker-compose command" - export USAGE_EXECUTE_COMMAND="Executes specified command in the container" export USAGE_FLAGS="Shows all breeze's flags" export USAGE_GENERATE_REQUIREMENTS="Generates pinned requirements for pip dependencies" export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv" @@ -1046,14 +996,14 @@ function prepare_usage() { export USAGE_STATIC_CHECK_ALL_FILES="Performs selected static check for all files" export USAGE_TOGGLE_SUPPRESS_CHEATSHEET="Toggles on/off cheatsheet" export USAGE_TOGGLE_SUPPRESS_ASCIIART="Toggles on/off asciiart" - export USAGE_TEST_TARGET="Runs selected test target in the container" + export USAGE_TESTS="Runs selected tests in the container" export USAGE_HELP="Shows this help message" export USAGE_HELP_ALL="Shows detailed help for all commands and flags" # shellcheck disable=SC2089 DETAILED_USAGE_SHELL=" -${CMDNAME} shell [FLAGS] -- <EXTRA_ARGS> +${CMDNAME} shell [FLAGS] [-- <EXTRA_ARGS>] This is default subcommand if no subcommand is used. @@ -1070,13 +1020,18 @@ ${CMDNAME} shell [FLAGS] -- <EXTRA_ARGS> and webserver ports are forwarded to appropriate database/webserver so that you can connect to it from your host environment. + You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is + especially useful to pass bash options, for example -c to execute command: + + '${CMDNAME} shell -- -c \"ls -la\"' + Flags: $(flag_footer) " # shellcheck disable=SC2090 export DETAILED_USAGE_SHELL export DETAILED_USAGE_EXEC=" -${CMDNAME} exec +${CMDNAME} exec [-- <EXTRA_ARGS>] Execs into interactive shell to an already running container. The container mus be started already by breeze shell command. If you are not familiar with tmux, this is the best @@ -1128,7 +1083,7 @@ $(flag_verbosity) export DETAILED_USAGE_CLEANUP_IMAGE # shellcheck disable=SC2089 DETAILED_USAGE_DOCKER_COMPOSE=" -${CMDNAME} docker-compose [FLAGS] COMMAND -- <EXTRA_ARGS> +${CMDNAME} docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>] Run docker-compose command instead of entering the environment. Use 'help' as command to see available commands. The <EXTRA_ARGS> passed after -- are treated @@ -1143,25 +1098,6 @@ $(flag_verbosity) " # shellcheck disable=SC2090 export DETAILED_USAGE_DOCKER_COMPOSE - # shellcheck disable=SC2089 - DETAILED_USAGE_EXECUTE_COMMAND=" -${CMDNAME} execute-command [FLAGS] COMMAND -- <EXTRA_ARGS> - - Run chosen command instead of entering the environment. The command is run using - 'bash -c \"<command with args>\" if you need to pass arguments to your command, you need - to pass them together with command surrounded with \" or '. Alternatively you can - pass arguments as <EXTRA_ARGS> passed after --. For example: - - '${CMDNAME} execute-command \"ls -la\"' or - '${CMDNAME} execute-command ls -- --la' - -Flags: -$(flag_airflow_variants) -$(flag_backend_variants) -$(flag_verbosity) -" - # shellcheck disable=SC2090 - export DETAILED_USAGE_EXECUTE_COMMAND export DETAILED_USAGE_FLAGS=" Explains in detail all the flags that can be used with breeze. " @@ -1199,7 +1135,7 @@ $(flag_airflow_variants) ${CMDNAME} push_image [FLAGS] Pushes images to docker registry. You can push the images to DockerHub registry (default) - or to the GitHub cache registry (if --registry-cache flag is used). + or to the GitHub registry (if --github-registry flag is used). For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify the repository to push to. For GitHub repository --github-organisation and --github-repo @@ -1212,8 +1148,8 @@ ${CMDNAME} push_image [FLAGS] '${CMDNAME} push-image' or '${CMDNAME} push-image --dockerhub-user user' to push to your private registry or '${CMDNAME} push-image --production-image' - to push production image or - '${CMDNAME} push-image --registry-cache' - to push to GitHub cache or - '${CMDNAME} push-image --registry-cache --github-organisation org' - for other organisation + '${CMDNAME} push-image --github-registry' - to push to GitHub image registry or + '${CMDNAME} push-image --github-registry --github-organisation org' - for other organisation Flags: $(flag_pull_push_docker_images) @@ -1264,7 +1200,7 @@ $(flag_footer) " export DETAILED_USAGE_RESTART export DETAILED_USAGE_STATIC_CHECK=" -${CMDNAME} static-check [FLAGS] STATIC_CHECK +${CMDNAME} static-check [FLAGS] STATIC_CHECK [-- <EXTRA_ARGS>] Run selected static checks for currently changed files. You should specify static check that you would like to run or 'all' to run all checks. One of: @@ -1276,44 +1212,29 @@ ${FORMATTED_STATIC_CHECKS} '${CMDNAME} static-check mypy' or '${CMDNAME} static-check mypy -- --files tests/core.py' + '${CMDNAME} static-check mypy -- --all-files' You can see all the options by adding --help EXTRA_ARG: '${CMDNAME} static-check mypy -- --help' " - export DETAILED_USAGE_STATIC_CHECK_ALL_FILES=" -${CMDNAME} static-check-all [FLAGS] STATIC_CHECK - - Run selected static checks for all applicable files. You should specify static check that - you would like to run or 'all' to run all checks. One of: - -${FORMATTED_STATIC_CHECKS} - - You can pass extra arguments including options to the pre-commit framework as - <EXTRA_ARGS> passed after --. For example: - - '${CMDNAME} static-check-all-files mypy' or - '${CMDNAME} static-check-all-files mypy -- --verbose' - - You can see all the options by adding --help EXTRA_ARG: - - '${CMDNAME} static-check-all-files mypy -- --help' -" # shellcheck disable=SC2089 - DETAILED_USAGE_TEST_TARGET=" -${CMDNAME} test-target [FLAGS] TEST_TARGET -- <EXTRA_ARGS> + DETAILED_USAGE_TESTS=" +${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>] Run the specified unit test target. There might be multiple targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated - as additional options passed to pytest. For example: + as additional options passed to pytest. You can pass 'tests' as target to + run all tests. For example: - '${CMDNAME} test-target tests/test_core.py -- --logging-level=DEBUG' + '${CMDNAME} tests tests/test_core.py -- --logging-level=DEBUG' + '${CMDNAME} tests tests Flags: $(flag_footer) " # shellcheck disable=SC2090 - export DETAILED_USAGE_TEST_TARGET + export DETAILED_USAGE_TESTS export DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET=" ${CMDNAME} toggle-suppress-cheatsheet @@ -1412,6 +1333,7 @@ function flag_airflow_variants() { echo " -p, --python <PYTHON_MAJOR_MINOR_VERSION> Python version used for the image. This is always major/minor version. + One of: ${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS} @@ -1610,10 +1532,10 @@ function flag_pull_push_docker_images() { -H, --dockerhub-repo DockerHub repository used to pull, push, build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_REPO:=}. --c, --registry-cache - If registry cache is enabled, pulls and pushes are done from the registry cache in github. - You need to be logged in to the registry in order to be able to pull/push from it and you - need to be committer to push to airflow registry. +-c, --github-registry + If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not + DockerHub. You need to be logged in to the registry in order to be able to pull/push from it + and you need to be committer to push to Apache Airflow' GitHub registry. -G, --github-organisation GitHub organisation used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_ORGANISATION:=}. @@ -1764,25 +1686,6 @@ function print_cheatsheet() { echo echo " Airflow Breeze CHEATSHEET" echo - print_line - echo - echo - print_line - echo - echo " Bash scripts to run commands quickly:" - echo - echo " * Enter the CI environment : ${BUILD_CACHE_DIR}/${LAST_DC_RUN_CI_FILE}" - echo " * Enter the production environment : ${BUILD_CACHE_DIR}/${LAST_DC_RUN_PROD_FILE}" - echo " * Run command in CI environment : ${BUILD_CACHE_DIR}/${LAST_DC_RUN_CI_FILE} "\ - "[command with args] [bash options]" - echo " * Run tests in CI environment : ${BUILD_CACHE_DIR}/${LAST_DC_TEST_CI_FILE} "\ - "[test target] [pytest options]" - echo " * Run docker-compose CI command : ${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE} "\ - "[docker compose command] [docker-compose options]" - echo " * Run docker-compose production cmd : ${BUILD_CACHE_DIR}/${LAST_DC_PROD_FILE} "\ - "[docker compose command] [docker-compose options]" - echo - set +e if ! command -v breeze; then print_line @@ -1890,7 +1793,7 @@ function run_static_checks { # command chosen function run_build_command { case "${COMMAND_TO_RUN}" in - run_tests|run_docker_compose|run_in_bash) + run_tests|run_docker_compose) prepare_ci_build rebuild_ci_image_if_needed ;; @@ -1965,9 +1868,9 @@ function run_breeze_command { case "${COMMAND_TO_RUN}" in enter_breeze) if [[ ${PRODUCTION_IMAGE} == "true" ]]; then - "${BUILD_CACHE_DIR}/${LAST_DC_RUN_PROD_FILE}" + "${BUILD_CACHE_DIR}/${LAST_DC_PROD_FILE}" run --service-ports --rm airflow "${@}" else - "${BUILD_CACHE_DIR}/${LAST_DC_RUN_CI_FILE}" + "${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE}" run --service-ports --rm airflow "${@}" fi ;; run_exec) @@ -1983,10 +1886,10 @@ function run_breeze_command { : "${AIRFLOW_TESTING_CONTAINER:?"ERROR! Breeze must be running in order to exec into running container"}" set -e docker exec -it "${AIRFLOW_TESTING_CONTAINER}" \ - "/opt/airflow/scripts/ci/in_container/entrypoint_exec.sh" + "/opt/airflow/scripts/ci/in_container/entrypoint_exec.sh" "${@}" ;; run_tests) - "${BUILD_CACHE_DIR}/${LAST_DC_TEST_CI_FILE}" "\"${TEST_TARGET}\"" "$@" + "${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE}" run --service-ports --rm airflow "$@" ;; run_docker_compose) set +u @@ -1998,12 +1901,9 @@ function run_breeze_command { "${DC_FILE}" "${DOCKER_COMPOSE_COMMAND}" "${EXTRA_DC_OPTIONS[@]}" "$@" set -u ;; - run_in_bash) - "${BUILD_CACHE_DIR}/${LAST_DC_RUN_CI_FILE}" "${RUN_COMMAND}" "$@" - ;; perform_static_checks) make_sure_precommit_is_installed - run_static_checks "$@" + run_static_checks "${@}" ;; build_image) ;; diff --git a/breeze-complete b/breeze-complete index 6038014..6bde8fa 100644 --- a/breeze-complete +++ b/breeze-complete @@ -99,7 +99,7 @@ kubernetes-mode: kubernetes-version: skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset verbose assume-yes assume-no assume-quit forward-credentials force-build-images force-pull-images production-image extras: force-clean-images use-local-cache -dockerhub-user: dockerhub-repo: registry-cache github-organisation: github-repo: +dockerhub-user: dockerhub-repo: github-registry github-organisation: github-repo: postgres-version: mysql-version: additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: " @@ -113,7 +113,6 @@ exec generate-requirements push-image initialize-local-virtualenv -kind-cluster setup-autocomplete stop restart @@ -122,11 +121,9 @@ toggle-suppress-asciiart" export BREEZE_EXTRA_ARG_COMMANDS=" docker-compose -execute-command kind-cluster static-check -static-check-all-files -test-target" +tests" export BREEZE_HELP_COMMANDS=" flags diff --git a/scripts/ci/ci_run_airflow_testing.sh b/scripts/ci/ci_run_airflow_testing.sh index ad24a77..7b69a36 100755 --- a/scripts/ci/ci_run_airflow_testing.sh +++ b/scripts/ci/ci_run_airflow_testing.sh @@ -25,11 +25,7 @@ function run_airflow_testing_in_docker() { -f "${MY_DIR}/docker-compose/backend-${BACKEND}.yml" \ "${INTEGRATIONS[@]}" \ "${DOCKER_COMPOSE_LOCAL[@]}" \ - run airflow \ - '/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"' \ - /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}" - # Note the command is there twice (!) because it is passed via bash -c - # and bash -c starts passing parameters from $0. TODO: fixme + run airflow "${@}" set -u } diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml index 34e31ff..0feea60 100644 --- a/scripts/ci/docker-compose/base.yml +++ b/scripts/ci/docker-compose/base.yml @@ -19,16 +19,9 @@ version: "2.2" services: airflow: image: ${AIRFLOW_IMAGE} - init: true - entrypoint: ["/bin/bash", "-c"] environment: - USER=root - ADDITIONAL_PATH=~/.local/bin - - HADOOP_DISTRO=cdh - - HADOOP_HOME=/opt/hadoop-cdh - - HADOOP_OPTS=-D/opt/krb5.conf - - HIVE_HOME=/opt/hive - - MINICLUSTER_HOME=/opt/minicluster - CELERY_BROKER_URLS=amqp://guest:guest@rabbitmq:5672,redis://redis:6379/0 - BACKEND - CI diff --git a/scripts/ci/docker-compose/local-prod.yml b/scripts/ci/docker-compose/local-prod.yml index 6342d33..4ad5d7e 100644 --- a/scripts/ci/docker-compose/local-prod.yml +++ b/scripts/ci/docker-compose/local-prod.yml @@ -31,7 +31,7 @@ services: - ../../../.kube:/root/.kube:cached - ../../../files:/files:cached - ../../../dist:/dist:cached - - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../scripts/prod/entrypoint_prod.sh:/entrypoint:cached - ../../../setup.cfg:/opt/airflow/setup.cfg:cached - ../../../setup.py:/opt/airflow/setup.py:cached - ../../../tests:/opt/airflow/tests:cached diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml index ff88c6c..822d49d 100644 --- a/scripts/ci/docker-compose/local.yml +++ b/scripts/ci/docker-compose/local.yml @@ -18,6 +18,8 @@ version: "2.2" services: airflow: + stdin_open: true # docker run -i + tty: true # docker run -t # We need to mount files an directories individually because some files # such apache_airflow.egg-info should not be mounted from host # we only mount those files that it makes sense to edit while developing @@ -49,7 +51,7 @@ services: - ../../../pytest.ini:/opt/airflow/pytest.ini:cached - ../../../requirements:/opt/airflow/requirements:cached - ../../../scripts:/opt/airflow/scripts:cached - - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint:cached - ../../../setup.cfg:/opt/airflow/setup.cfg:cached - ../../../setup.py:/opt/airflow/setup.py:cached - ../../../tests:/opt/airflow/tests:cached diff --git a/scripts/ci/in_container/entrypoint_ci.sh b/scripts/ci/in_container/entrypoint_ci.sh index 2a7535c..1a753cf 100755 --- a/scripts/ci/in_container/entrypoint_ci.sh +++ b/scripts/ci/in_container/entrypoint_ci.sh @@ -20,7 +20,7 @@ if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then fi # shellcheck source=scripts/ci/in_container/_in_container_script_init.sh -. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh" +. /opt/airflow/scripts/ci/in_container/_in_container_script_init.sh AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../.." || exit 1; pwd) @@ -41,8 +41,6 @@ fi echo -ARGS=( "$@" ) - RUN_TESTS=${RUN_TESTS:="true"} INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION:=""}" @@ -86,15 +84,6 @@ fi export RUN_AIRFLOW_1_10=${RUN_AIRFLOW_1_10:="false"} -export HADOOP_DISTRO="${HADOOP_DISTRO:="cdh"}" -export HADOOP_HOME="${HADOOP_HOME:="/opt/hadoop-cdh"}" - -if [[ ${VERBOSE} == "true" ]]; then - echo - echo "Using ${HADOOP_DISTRO} distribution of Hadoop from ${HADOOP_HOME}" - echo -fi - # Added to have run-tests on path export PATH=${PATH}:${AIRFLOW_SOURCES} @@ -134,10 +123,6 @@ if [[ ${INTEGRATION_KERBEROS:="false"} == "true" ]]; then fi -# Start MiniCluster -java -cp "/opt/minicluster-1.1-SNAPSHOT/*" com.ing.minicluster.MiniCluster \ - >"${AIRFLOW_HOME}/logs/minicluster.log" 2>&1 & - # Set up ssh keys echo 'yes' | ssh-keygen -t rsa -C [email protected] -m PEM -P '' -f ~/.ssh/id_rsa \ >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1 @@ -160,25 +145,13 @@ ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null # shellcheck source=scripts/ci/in_container/configure_environment.sh . "${MY_DIR}/configure_environment.sh" -if [[ ${CI:=} == "true" && ${RUN_TESTS} == "true" ]] ; then - echo - echo " !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" - echo " Setting default parallellism to 2 because we can run out of memory during tests on CI" - echo " !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" - echo - export AIRFLOW__CORE__PARALELLISM=2 -fi +cd "${AIRFLOW_SOURCES}" set +u # If we do not want to run tests, we simply drop into bash -if [[ "${RUN_TESTS}" == "false" ]]; then - if [[ ${#ARGS} == 0 ]]; then - exec /bin/bash - else - exec /bin/bash -c "$(printf "%q " "${ARGS[@]}")" - fi +if [[ "${RUN_TESTS:=false}" != "true" ]]; then + exec /bin/bash "${@}" fi - set -u if [[ "${CI}" == "true" ]]; then @@ -199,10 +172,11 @@ else CI_ARGS=() fi -TESTS_TO_RUN="tests/" +declare -a TESTS_TO_RUN +TESTS_TO_RUN=("tests") if [[ ${#@} -gt 0 && -n "$1" ]]; then - TESTS_TO_RUN="$1" + TESTS_TO_RUN=("${@}") fi if [[ -n ${RUN_INTEGRATION_TESTS:=""} ]]; then @@ -227,7 +201,7 @@ elif [[ ${ONLY_RUN_QUARANTINED_TESTS:=""} == "true" ]]; then "--timeout" "90") fi -ARGS=("${CI_ARGS[@]}" "${TESTS_TO_RUN}") +ARGS=("${CI_ARGS[@]}" "${TESTS_TO_RUN[@]}") if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then "${MY_DIR}/run_system_tests.sh" "${ARGS[@]}" diff --git a/scripts/ci/in_container/entrypoint_exec.sh b/scripts/ci/in_container/entrypoint_exec.sh index d675d91..cfee03a 100755 --- a/scripts/ci/in_container/entrypoint_exec.sh +++ b/scripts/ci/in_container/entrypoint_exec.sh @@ -18,4 +18,4 @@ # shellcheck source=scripts/ci/in_container/configure_environment.sh . "$( dirname "${BASH_SOURCE[0]}" )/configure_environment.sh" -exec /bin/bash +exec /bin/bash "${@}" diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/in_container/run_ci_tests.sh index fe5a574..dc86cf0 100755 --- a/scripts/ci/in_container/run_ci_tests.sh +++ b/scripts/ci/in_container/run_ci_tests.sh @@ -18,15 +18,12 @@ # shellcheck source=scripts/ci/in_container/_in_container_script_init.sh . "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh" -# any argument received is overriding the default nose execution arguments: -PYTEST_ARGS=( "$@" ) - echo -echo "Starting the tests with those pytest arguments: ${PYTEST_ARGS[*]}" +echo "Starting the tests with those pytest arguments:" "${@}" echo set +e -pytest "${PYTEST_ARGS[@]}" +pytest "${@}" RES=$? @@ -36,7 +33,7 @@ if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then bash <(curl -s https://codecov.io/bash) fi -if [[ ${CI} == "true" ]]; then +if [[ ${CI:=} == "true" ]]; then send_airflow_logs_to_file_io fi diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh index a54d6d8..fcb2cc8 100644 --- a/scripts/ci/libraries/_build_images.sh +++ b/scripts/ci/libraries/_build_images.sh @@ -23,10 +23,9 @@ function add_build_args_for_remote_install() { # entrypoint is used as AIRFLOW_SOURCES_FROM/TO in order to avoid costly copying of all sources of # Airflow - those are not needed for remote install at all. Entrypoint is later overwritten by - # ENTRYPOINT_FILE - downloaded entrypoint.sh so this is only for the purpose of iteration on Dockerfile EXTRA_DOCKER_PROD_BUILD_FLAGS+=( - "--build-arg" "AIRFLOW_SOURCES_FROM=entrypoint.sh" - "--build-arg" "AIRFLOW_SOURCES_TO=/entrypoint" + "--build-arg" "AIRFLOW_SOURCES_FROM=empty" + "--build-arg" "AIRFLOW_SOURCES_TO=/empty" ) if [[ ${AIRFLOW_VERSION} =~ [^0-9]*1[^0-9]*10[^0-9]([0-9]*) ]]; then # All types of references/versions match this regexp for 1.10 series @@ -36,21 +35,20 @@ function add_build_args_for_remote_install() { # This is only for 1.10.0 - 1.10.9 EXTRA_DOCKER_PROD_BUILD_FLAGS+=( "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"; - "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"; ) else EXTRA_DOCKER_PROD_BUILD_FLAGS+=( # For 1.10.10+ and v1-10-test it's ok to use AIRFLOW_VERSION as reference "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"; - "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"; ) fi + AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v1-10-test" else # For all other (master, 2.0+) we just match ${AIRFLOW_VERSION} EXTRA_DOCKER_PROD_BUILD_FLAGS+=( "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"; - "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"; ) + AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="master" fi } @@ -205,10 +203,10 @@ function get_local_image_info() { TMP_MANIFEST_LOCAL_SHA=$(mktemp) set +e # Remove the container just in case - verbose_docker rm --force "local-airflow-manifest" >/dev/null 2>&1 + verbose_docker_hide_output_on_success rm --force "local-airflow-manifest" # Create manifest from the local manifest image - if ! verbose_docker create --name "local-airflow-manifest" \ - "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" >/dev/null 2>&1 ; then + if ! verbose_docker_hide_output_on_success create --name "local-airflow-manifest" \ + "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" >>"${OUTPUT_LOG}" 2>&1 ; then echo echo "Local manifest image not available" echo @@ -217,9 +215,10 @@ function get_local_image_info() { fi set -e # Create manifest from the local manifest image - verbose_docker cp "local-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_LOCAL_JSON}" >/dev/null 2>&1 + verbose_docker_hide_output_on_success cp "local-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" \ + "${TMP_MANIFEST_LOCAL_JSON}" >>"${OUTPUT_LOG}" 2>&1 sed 's/ *//g' "${TMP_MANIFEST_LOCAL_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_LOCAL_SHA}" - verbose_docker rm --force "local-airflow-manifest" >/dev/null 2>&1 + verbose_docker_hide_output_on_success rm --force "local-airflow-manifest" >>"${OUTPUT_LOG}" 2>&1 } # @@ -233,7 +232,7 @@ function get_local_image_info() { function get_remote_image_info() { set +e # Pull remote manifest image - if ! verbose_docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null; then + if ! verbose_docker_hide_output_on_success pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"; then echo echo "Remote docker registry unreachable" echo @@ -250,12 +249,14 @@ function get_remote_image_info() { TMP_MANIFEST_REMOTE_JSON=$(mktemp) TMP_MANIFEST_REMOTE_SHA=$(mktemp) # Create container out of the manifest image without running it - verbose_docker create --cidfile "${TMP_CONTAINER_ID}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" + verbose_docker_hide_output_on_success create --cidfile "${TMP_CONTAINER_ID}" \ + "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" # Extract manifest and store it in local file - verbose_docker cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}" + verbose_docker_hide_output_on_success cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" \ + "${TMP_MANIFEST_REMOTE_JSON}" # Filter everything except SHAs of image layers sed 's/ *//g' "${TMP_MANIFEST_REMOTE_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_REMOTE_SHA}" - verbose_docker rm --force "$( cat "${TMP_CONTAINER_ID}")" + verbose_docker_hide_output_on_success rm --force "$( cat "${TMP_CONTAINER_ID}")" } # The Number determines the cut-off between local building time and pull + build time. @@ -273,7 +274,7 @@ function get_remote_image_info() { # Note that this only matters if you have any of the important files changed since the last build # of your image such as Dockerfile.ci, setup.py etc. # -MAGIC_CUT_OFF_NUMBER_OF_LAYERS=34 +MAGIC_CUT_OFF_NUMBER_OF_LAYERS=36 # Compares layers from both remote and local image and set FORCE_PULL_IMAGES to true in case # More than the last NN layers are different. @@ -315,11 +316,11 @@ function print_build_info() { # Prepares all variables needed by the CI build. Depending on the configuration used (python version # DockerHub user etc. the variables are set so that other functions can use those variables. function prepare_ci_build() { - export AIRFLOW_CI_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}-ci" + export AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci" export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest" export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest" export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}" - if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then + if [[ ${USE_GITHUB_REGISTRY="false"} == "true" ]]; then if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ --username "${CACHE_REGISTRY_USERNAME}" \ @@ -334,7 +335,7 @@ function prepare_ci_build() { export CACHED_PYTHON_BASE_IMAGE="" fi export AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}" - export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}-ci" + export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci" export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster" export BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}" if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then @@ -569,10 +570,31 @@ Docker building ${AIRFLOW_CI_IMAGE}. # Prepares all variables needed by the CI build. Depending on the configuration used (python version # DockerHub user etc. the variables are set so that other functions can use those variables. function prepare_prod_build() { - export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}" + if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then + # When --install-airflow-reference is used then the image is build from github tag + EXTRA_DOCKER_PROD_BUILD_FLAGS=( + "--build-arg" "AIRFLOW_INSTALL_SOURCES=https://github.com/apache/airflow/archive/${INSTALL_AIRFLOW_REFERENCE}.tar.gz#egg=apache-airflow"; + ) + export AIRFLOW_VERSION="${INSTALL_AIRFLOW_REFERENCE}" + add_build_args_for_remote_install + elif [[ "${INSTALL_AIRFLOW_VERSION:=}" != "" ]]; then + # When --install-airflow-version is used then the image is build from PIP package + EXTRA_DOCKER_PROD_BUILD_FLAGS=( + "--build-arg" "AIRFLOW_INSTALL_SOURCES=apache-airflow" + "--build-arg" "AIRFLOW_INSTALL_VERSION===${INSTALL_AIRFLOW_VERSION}" + ) + export AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}" + add_build_args_for_remote_install + else + # When no airflow version/reference is specified, production image is built from local sources + EXTRA_DOCKER_PROD_BUILD_FLAGS=( + ) + fi + + export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}" export AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build" export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}" - export AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}" + export AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}" export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster" if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then export DEFAULT_IMAGE="${AIRFLOW_PROD_IMAGE_DEFAULT}" @@ -588,7 +610,7 @@ function prepare_prod_build() { export ADDITIONAL_RUNTIME_DEPS="${ADDITIONAL_RUNTIME_DEPS:=""}" export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}" - if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then + if [[ ${USE_GITHUB_REGISTRY="false"} == "true" ]]; then if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then echo "${CACHE_REGISTRY_PASSWORD}" | docker login \ --username "${CACHE_REGISTRY_USERNAME}" \ @@ -610,26 +632,8 @@ function prepare_prod_build() { AIRFLOW_KUBERNETES_IMAGE_TAG=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 2 -d ":") export AIRFLOW_KUBERNETES_IMAGE_TAG - if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then - # When --install-airflow-reference is used then the image is build from github tag - EXTRA_DOCKER_PROD_BUILD_FLAGS=( - "--build-arg" "AIRFLOW_INSTALL_SOURCES=https://github.com/apache/airflow/archive/${INSTALL_AIRFLOW_REFERENCE}.tar.gz#egg=apache-airflow"; - ) - export AIRFLOW_VERSION="${INSTALL_AIRFLOW_REFERENCE}" - add_build_args_for_remote_install - elif [[ "${INSTALL_AIRFLOW_VERSION:=}" != "" ]]; then - # When --install-airflow-version is used then the image is build from PIP package - EXTRA_DOCKER_PROD_BUILD_FLAGS=( - "--build-arg" "AIRFLOW_INSTALL_SOURCES=apache-airflow" - "--build-arg" "AIRFLOW_INSTALL_VERSION===${INSTALL_AIRFLOW_VERSION}" - ) - export AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}" - add_build_args_for_remote_install - else - # When no airflow version/reference is specified, production image is built from local sources - EXTRA_DOCKER_PROD_BUILD_FLAGS=( - ) - fi + AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="${BRANCH_NAME}" + go_to_airflow_sources } @@ -669,6 +673,7 @@ function build_prod_image() { --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \ --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ + --build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \ --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \ --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \ @@ -687,6 +692,7 @@ function build_prod_image() { --build-arg ADDITIONAL_DEV_DEPS="${ADDITIONAL_DEV_DEPS}" \ --build-arg ADDITIONAL_RUNTIME_DEPS="${ADDITIONAL_RUNTIME_DEPS}" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ + --build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \ --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \ "${DOCKER_CACHE_PROD_DIRECTIVE[@]}" \ -t "${AIRFLOW_PROD_IMAGE}" \ diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh index 6f66b0d..f70655b 100644 --- a/scripts/ci/libraries/_initialization.sh +++ b/scripts/ci/libraries/_initialization.sh @@ -46,6 +46,16 @@ function initialize_common_environment { # All the subsequent questions export LAST_FORCE_ANSWER_FILE="${BUILD_CACHE_DIR}/last_force_answer.sh" + # This folder is mounted to inside the container in /files folder. This is the way how + # We can exchange DAGs, scripts, packages etc with the container environment + export FILES_DIR="${AIRFLOW_SOURCES}/files" + # Temporary dir used well ... temporarily + export TMP_DIR="${AIRFLOW_SOURCES}/tmp" + + # Create those folders above in case they do not exist + mkdir -p "${TMP_DIR}" + mkdir -p "${FILES_DIR}" + # Create useful directories if not yet created mkdir -p "${AIRFLOW_SOURCES}/.mypy_cache" mkdir -p "${AIRFLOW_SOURCES}/logs" @@ -69,7 +79,7 @@ function initialize_common_environment { export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"} export GITHUB_REPO=${GITHUB_REPO:="airflow"} export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"} - export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"} + export USE_GITHUB_REGISTRY=${USE_GITHUB_REGISTRY:="false"} # Default port numbers for forwarded ports export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"} diff --git a/scripts/ci/libraries/_local_mounts.sh b/scripts/ci/libraries/_local_mounts.sh index 24a2eb3..127ebb3 100644 --- a/scripts/ci/libraries/_local_mounts.sh +++ b/scripts/ci/libraries/_local_mounts.sh @@ -47,7 +47,7 @@ function generate_local_mounts_list { "$prefix"pytest.ini:/opt/airflow/pytest.ini:cached "$prefix"requirements:/opt/airflow/requirements:cached "$prefix"scripts:/opt/airflow/scripts:cached - "$prefix"scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + "$prefix"scripts/ci/in_container/entrypoint_ci.sh:/entrypoint:cached "$prefix"setup.cfg:/opt/airflow/setup.cfg:cached "$prefix"setup.py:/opt/airflow/setup.py:cached "$prefix"tests:/opt/airflow/tests:cached diff --git a/scripts/ci/libraries/_md5sum.sh b/scripts/ci/libraries/_md5sum.sh index c9a254a..95e4478 100644 --- a/scripts/ci/libraries/_md5sum.sh +++ b/scripts/ci/libraries/_md5sum.sh @@ -24,7 +24,7 @@ function calculate_file_md5sum { local FILE="${1}" local MD5SUM - local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}" + local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${BRANCH_NAME}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}" mkdir -pv "${MD5SUM_CACHE_DIR}" MD5SUM=$(md5sum "${FILE}") local MD5SUM_FILE @@ -54,7 +54,7 @@ function calculate_file_md5sum { function move_file_md5sum { local FILE="${1}" local MD5SUM_FILE - local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}" + local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${BRANCH_NAME}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}" mkdir -pv "${MD5SUM_CACHE_DIR}" MD5SUM_FILE="${MD5SUM_CACHE_DIR}"/$(basename "${FILE}").md5sum local MD5SUM_FILE_NEW diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh index c09ff57..7b9d8a8 100644 --- a/scripts/ci/libraries/_push_pull_remove_images.sh +++ b/scripts/ci/libraries/_push_pull_remove_images.sh @@ -112,8 +112,8 @@ function pull_prod_images_if_needed() { fi # "Build" segment of production image pull_image_possibly_from_cache "${AIRFLOW_PROD_BUILD_IMAGE}" "${CACHED_AIRFLOW_PROD_BUILD_IMAGE}" - # Main segment of production image - pull_image_possibly_from_cache "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}" + # we never pull the main segment of production image - we always build it locally = this is + # usually very fast this way and it is much nicer for rebuilds and development fi } @@ -162,7 +162,6 @@ function push_prod_images() { if [[ -n ${DEFAULT_IMAGE:=""} && ${CACHED_AIRFLOW_PROD_IMAGE} == "" ]]; then verbose_docker push "${DEFAULT_IMAGE}" fi - # we do not need to push PYTHON base image here - they are already pushed in the CI push } diff --git a/scripts/ci/libraries/_start_end.sh b/scripts/ci/libraries/_start_end.sh index b50a405..4dcb150 100644 --- a/scripts/ci/libraries/_start_end.sh +++ b/scripts/ci/libraries/_start_end.sh @@ -54,6 +54,10 @@ function script_end { #shellcheck disable=2181 EXIT_CODE=$? if [[ ${EXIT_CODE} != 0 ]]; then + # Cat output log in case we exit with error + if [[ -f "${OUTPUT_LOG}" ]]; then + cat "${OUTPUT_LOG}" + fi print_info "###########################################################################################" print_info " EXITING WITH STATUS CODE ${EXIT_CODE}" print_info "###########################################################################################" diff --git a/scripts/ci/libraries/_verbosity.sh b/scripts/ci/libraries/_verbosity.sh index 9958635..b7a4c0a 100644 --- a/scripts/ci/libraries/_verbosity.sh +++ b/scripts/ci/libraries/_verbosity.sh @@ -35,6 +35,17 @@ function verbose_docker { docker "${@}" } +# In case "VERBOSE" is set to "true" (--verbose flag in Breeze) all docker commands run will be +# printed before execution +function verbose_docker_hide_output_on_success { + if [[ ${VERBOSE:="false"} == "true" && ${VERBOSE_COMMANDS:=} != "true" ]]; then + # do not print echo if VERBOSE_COMMAND is set (set -x does it already) + echo "docker" "${@}" + fi + docker "${@}" >>"${OUTPUT_LOG}" 2>&1 +} + + # Prints verbose information in case VERBOSE variable is set function print_info() { if [[ ${VERBOSE:="false"} == "true" ]]; then diff --git a/scripts/docker/entrypoint.sh b/scripts/docker/entrypoint.sh deleted file mode 100755 index 3d436e2..0000000 --- a/scripts/docker/entrypoint.sh +++ /dev/null @@ -1,110 +0,0 @@ -#!/usr/bin/env bash -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Might be empty -AIRFLOW_COMMAND="${1}" - -set -euo pipefail - -function verify_db_connection { - DB_URL="${1}" - - DB_CHECK_MAX_COUNT=${MAX_DB_CHECK_COUNT:=20} - DB_CHECK_SLEEP_TIME=${DB_CHECK_SLEEP_TIME:=3} - - local DETECTED_DB_BACKEND="" - local DETECTED_DB_HOST="" - local DETECTED_DB_PORT="" - - - if [[ ${DB_URL} != sqlite* ]]; then - # Auto-detect DB parameters - [[ ${DB_URL} =~ ([^:]*)://([^@/]*)@?([^/:]*):?([0-9]*)/([^\?]*)\??(.*) ]] && \ - DETECTED_DB_BACKEND=${BASH_REMATCH[1]} && - # Not used USER match - DETECTED_DB_HOST=${BASH_REMATCH[3]} && - DETECTED_DB_PORT=${BASH_REMATCH[4]} && - # Not used SCHEMA match - # Not used PARAMS match - - echo DB_BACKEND="${DB_BACKEND:=${DETECTED_DB_BACKEND}}" - - if [[ -z "${DETECTED_DB_PORT}" ]]; then - if [[ ${DB_BACKEND} == "postgres"* ]]; then - DETECTED_DB_PORT=5432 - elif [[ ${DB_BACKEND} == "mysql"* ]]; then - DETECTED_DB_PORT=3306 - fi - fi - - DETECTED_DB_HOST=${DETECTED_DB_HOST:="localhost"} - - # Allow the DB parameters to be overridden by environment variable - echo DB_HOST="${DB_HOST:=${DETECTED_DB_HOST}}" - echo DB_PORT="${DB_PORT:=${DETECTED_DB_PORT}}" - - while true - do - set +e - LAST_CHECK_RESULT=$(nc -zvv "${DB_HOST}" "${DB_PORT}" >/dev/null 2>&1) - RES=$? - set -e - if [[ ${RES} == 0 ]]; then - echo - break - else - echo -n "." - DB_CHECK_MAX_COUNT=$((DB_CHECK_MAX_COUNT-1)) - fi - if [[ ${DB_CHECK_MAX_COUNT} == 0 ]]; then - echo - echo "ERROR! Maximum number of retries (${DB_CHECK_MAX_COUNT}) reached while checking ${DB_BACKEND} db. Exiting" - echo - break - else - sleep "${DB_CHECK_SLEEP_TIME}" - fi - done - if [[ ${RES} != 0 ]]; then - echo " ERROR: ${BACKEND} db could not be reached!" - echo - echo "${LAST_CHECK_RESULT}" - echo - export EXIT_CODE=${RES} - fi - fi -} - -# if no DB configured - use sqlite db by default -AIRFLOW__CORE__SQL_ALCHEMY_CONN="${AIRFLOW__CORE__SQL_ALCHEMY_CONN:="sqlite:///${AIRFLOW_HOME}/airflow.db"}" - -verify_db_connection "${AIRFLOW__CORE__SQL_ALCHEMY_CONN}" - -AIRFLOW__CELERY__BROKER_URL=${AIRFLOW__CELERY__BROKER_URL:=} - -if [[ -n ${AIRFLOW__CELERY__BROKER_URL} ]] && \ - [[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|flower)$ ]]; then - verify_db_connection "${AIRFLOW__CELERY__BROKER_URL}" -fi - -if [[ ${AIRFLOW_COMMAND} == "" ]]; then - exec "/bin/bash" -fi - -# Run the command -exec airflow "${@}" diff --git a/scripts/ci/in_container/entrypoint_exec.sh b/scripts/prod/clean-logs.sh similarity index 64% copy from scripts/ci/in_container/entrypoint_exec.sh copy to scripts/prod/clean-logs.sh index d675d91..d05f8a8 100755 --- a/scripts/ci/in_container/entrypoint_exec.sh +++ b/scripts/prod/clean-logs.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash + # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -15,7 +16,22 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. -# shellcheck source=scripts/ci/in_container/configure_environment.sh -. "$( dirname "${BASH_SOURCE[0]}" )/configure_environment.sh" -exec /bin/bash +set -euo pipefail + +DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}" +RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}" + +trap "exit" INT TERM + +EVERY=$((15*60)) + +echo "Cleaning logs every $EVERY seconds" + +while true; do + seconds=$(( $(date -u +%s) % EVERY)) + [[ $seconds -lt 1 ]] || sleep $((EVERY - seconds)) + + echo "Trimming airflow logs to ${RETENTION} days." + find "${DIRECTORY}"/logs -mtime +"${RETENTION}" -name '*.log' -delete +done diff --git a/entrypoint.sh b/scripts/prod/entrypoint_prod.sh similarity index 100% rename from entrypoint.sh rename to scripts/prod/entrypoint_prod.sh
