[
https://issues.apache.org/jira/browse/AIRFLOW-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921452#comment-16921452
]
ASF GitHub Bot commented on AIRFLOW-5389:
-----------------------------------------
potiuk commented on pull request #5989: [AIRFLOW-5389] better organized scripts
for building CI docker deps
URL: https://github.com/apache/airflow/pull/5989
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Better organised/faster build for external dependencies in ci build
> -------------------------------------------------------------------
>
> Key: AIRFLOW-5389
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5389
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
> Affects Versions: 2.0.0, 1.10.5
> Reporter: Jarek Potiuk
> Priority: Major
>
> There are several entries with if's for CI image in the Dockerfile. They have
> weird syntax and they are slow (each if line is separately evaluated while
> Docker build).
> They can be moved to bash scripts and replaced with single RUN command (one
> per CI deps and one for main file). Then we will also use the power of
> shellcheck to verify those scripts.
> In the future, we also think about unifying the dockerfiles later one
> (production/CI/SLIM without multi-staging) and removing installation of
> hive/hadoop/minicluster to separate images. This might be first step to do so.
> Examples of the current code:
> CI deps:
> {code:java}
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
> mkdir -pv ${HADOOP_HOME} \
> && mkdir -pv ${HIVE_HOME} \
> && mkdir /tmp/minicluster \
> && mkdir -pv /user/hive/warehouse \
> && chmod -R 777 ${HIVE_HOME} \
> && chmod -R 777 /user/ \
> ;\
> fi
> # Install Hadoop
> # --absolute-names is a work around to avoid this issue
> https://github.com/docker/hub-feedback/issues/727
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
>
> HADOOP_URL=${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz
> \
> && HADOOP_TMP_FILE=/tmp/hadoop.tar.gz \
> && curl -sL ${HADOOP_URL} > ${HADOOP_TMP_FILE} \
> && tar xzf ${HADOOP_TMP_FILE} --absolute-names --strip-components 1 -C
> ${HADOOP_HOME} \
> && rm ${HADOOP_TMP_FILE} \
> ;\
> fi{code}
>
> main:
> {code:java}
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
> chmod +x "${HOME}/run-tests-complete" "${HOME}/run-tests" && \
> chown "${AIRFLOW_USER}.${AIRFLOW_USER}" "${HOME}/.bashrc" \
> "${HOME}/run-tests-complete" "${HOME}/run-tests"; \
> fi
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
