This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 55d15ae6ee71b839697d752bd8626c01c8b3ddc0 Author: zikun <[email protected]> AuthorDate: Wed Jun 10 05:05:43 2020 +0800 Support additional apt dependencies (#9189) * Add ADDITONAL_DEV_DEPS and ADDITONAL_RUNTIME_DEPS * Add examples for additional apt dev and runtime dependencies * Update comment * Fix typo (cherry picked from commit 82c8343ab6294168104cb2f25018656b681d2de9) --- Dockerfile | 12 ++++++++++-- Dockerfile.ci | 10 +++++++++- IMAGES.rst | 44 ++++++++++++++++++++++++++++++++++++++++++++ docs/concepts.rst | 2 +- 4 files changed, 64 insertions(+), 4 deletions(-) diff --git a/Dockerfile b/Dockerfile index ac27823..7c722cf 100644 --- a/Dockerfile +++ b/Dockerfile @@ -79,7 +79,10 @@ RUN apt-get update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* -# Install basic apt dependencies +ARG ADDITIONAL_DEV_DEPS="" +ENV ADDITIONAL_DEV_DEPS=${ADDITIONAL_DEV_DEPS} + +# Install basic and additional apt dependencies RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \ && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \ && echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \ @@ -121,6 +124,7 @@ RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \ unixodbc \ unixodbc-dev \ yarn \ + ${ADDITIONAL_DEV_DEPS} \ && apt-get autoremove -yqq --purge \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* @@ -242,13 +246,16 @@ ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} ARG AIRFLOW_VERSION ENV AIRFLOW_VERSION=${AIRFLOW_VERSION} +ARG ADDITIONAL_RUNTIME_DEPS="" +ENV ADDITIONAL_RUNTIME_DEPS=${ADDITIONAL_RUNTIME_DEPS} + # Make sure noninteractive debian install is used and language variables set ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \ LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 # Note missing man directories on debian-buster # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199 -# Install basic apt dependencies +# Install basic and additional apt dependencies RUN mkdir -pv /usr/share/man/man1 \ && mkdir -pv /usr/share/man/man7 \ && apt-get update \ @@ -277,6 +284,7 @@ RUN mkdir -pv /usr/share/man/man1 \ sqlite3 \ sudo \ unixodbc \ + ${ADDITIONAL_RUNTIME_DEPS} \ && apt-get autoremove -yqq --purge \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* diff --git a/Dockerfile.ci b/Dockerfile.ci index 8051431..1549214 100644 --- a/Dockerfile.ci +++ b/Dockerfile.ci @@ -54,7 +54,10 @@ RUN apt-get update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* -# Install basic apt dependencies +ARG ADDITIONAL_DEV_DEPS="" +ENV ADDITIONAL_DEV_DEPS=${ADDITIONAL_DEV_DEPS} + +# Install basic and additional apt dependencies RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \ && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \ && echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \ @@ -83,6 +86,7 @@ RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \ sasl2-bin \ sudo \ yarn \ + ${ADDITIONAL_DEV_DEPS} \ && apt-get autoremove -yqq --purge \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* @@ -115,6 +119,9 @@ RUN adduser airflow \ && echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow \ && chmod 0440 /etc/sudoers.d/airflow +ARG ADDITIONAL_RUNTIME_DEPS="" +ENV ADDITIONAL_RUNTIME_DEPS=${ADDITIONAL_RUNTIME_DEPS} + # Note missing man directories on debian-buster # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199 RUN mkdir -pv /usr/share/man/man1 \ @@ -142,6 +149,7 @@ RUN mkdir -pv /usr/share/man/man1 \ tmux \ unzip \ vim \ + ${ADDITIONAL_RUNTIME_DEPS} \ && apt-get autoremove -yqq --purge \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* diff --git a/IMAGES.rst b/IMAGES.rst index 0a34140..0d4bd8c 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -203,6 +203,12 @@ The following build arguments (``--build-arg`` in docker build command) can be u | ``ADDITIONAL_PYTHON_DEPS`` | \```\` | additional python dependencies to | | | | install | +------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_RUNTIME_DEPS`` | ```` | additional apt runtime dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ Here are some examples of how CI images can built manually. CI is always built from local sources. @@ -236,6 +242,20 @@ This builds the CI image in version 3.6 with "mssql" additional package added. docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_PYTHON_DEPS="mssql" +This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev dependencies added. + +.. code-block:: + + docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_DEV_DEPS="gcc g++" + +This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added. + +.. code-block:: + + docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" + Production images @@ -277,6 +297,12 @@ The following build arguments (``--build-arg`` in docker build command) can be u | ``ADDITIONAL_PYTHON_DEPS`` | ```` | Optional python packages to extend | | | | the image with some extra dependencies | +------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_RUNTIME_DEPS`` | ```` | additional apt runtime dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_HOME`` | ``/opt/airflow`` | Airflow’s HOME (that’s where logs and | | | | sqlite databases are stored) | +------------------------------------------+------------------------------------------+------------------------------------------+ @@ -402,6 +428,24 @@ additional python dependencies. --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client" +This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and +additional apt dev and runtime dependencies. + +.. code-block:: + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \ + --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"; \ + --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"; \ + --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \ + --build-arg AIRFLOW_SOURCES_TO="/entrypoint" \ + --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" + --build-arg ADDITIONAL_DEV_DEPS="gcc g++" + --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" + Image manifests --------------- diff --git a/docs/concepts.rst b/docs/concepts.rst index 3a9c4c2..603a729 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -522,7 +522,7 @@ with the same ``conn_id``, the :py:meth:`~airflow.hooks.base_hook.BaseHook.get_c provide basic load balancing and fault tolerance, when used in conjunction with retries. Airflow also provides a mechanism to store connections outside the database, e.g. in :ref:`environment variables <environment_variables_secrets_backend>`. -Additonal sources may be enabled, e.g. :ref:`AWS SSM Parameter Store <ssm_parameter_store_secrets>`, or you may +Additional sources may be enabled, e.g. :ref:`AWS SSM Parameter Store <ssm_parameter_store_secrets>`, or you may :ref:`roll your own secrets backend <roll_your_own_secrets_backend>`. Many hooks have a default ``conn_id``, where operators using that hook do not
