Jarek Potiuk created AIRFLOW-5389:
-------------------------------------
Summary: Better organised/faster build for external dependencies
in ci build
Key: AIRFLOW-5389
URL: https://issues.apache.org/jira/browse/AIRFLOW-5389
Project: Apache Airflow
Issue Type: Improvement
Components: ci
Affects Versions: 2.0.0, 1.10.5
Reporter: Jarek Potiuk
There are several entries with if's for CI image in the Dockerfile. They have
weird syntax and they are slow (each if line is separately evaluated while
Docker build).
They can be moved to bash scripts and replaced with single RUN command (one per
CI deps and one for main file). Then we will also use the power of shellcheck
to verify those scripts.
In the future, we also think about unifying the dockerfiles later one
(production/CI/SLIM without multi-staging) and removing installation of
hive/hadoop/minicluster to separate images. This might be first step to do so.
Examples of the current code:
CI deps:
{code:java}
RUN \
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
mkdir -pv ${HADOOP_HOME} \
&& mkdir -pv ${HIVE_HOME} \
&& mkdir /tmp/minicluster \
&& mkdir -pv /user/hive/warehouse \
&& chmod -R 777 ${HIVE_HOME} \
&& chmod -R 777 /user/ \
;\
fi
# Install Hadoop
# --absolute-names is a work around to avoid this issue
https://github.com/docker/hub-feedback/issues/727
RUN \
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
HADOOP_URL=${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz
\
&& HADOOP_TMP_FILE=/tmp/hadoop.tar.gz \
&& curl -sL ${HADOOP_URL} > ${HADOOP_TMP_FILE} \
&& tar xzf ${HADOOP_TMP_FILE} --absolute-names --strip-components 1 -C
${HADOOP_HOME} \
&& rm ${HADOOP_TMP_FILE} \
;\
fi{code}
main:
{code:java}
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
chmod +x "${HOME}/run-tests-complete" "${HOME}/run-tests" && \
chown "${AIRFLOW_USER}.${AIRFLOW_USER}" "${HOME}/.bashrc" \
"${HOME}/run-tests-complete" "${HOME}/run-tests"; \
fi
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
