potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r299598808
##########
File path: scripts/ci/kubernetes/kube/deploy.sh
##########
@@ -123,72 +123,78 @@ ${SED_COMMAND} -i "s|{{AIRFLOW_TAG}}|$AIRFLOW_TAG|g"
${BUILD_DIRNAME}/airflow.ya
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{INIT_DAGS_VOLUME_NAME}}|$INIT_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
-${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
+${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g" \
+ ${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} "s|{{CONFIGMAP_DAGS_FOLDER}}|$CONFIGMAP_DAGS_FOLDER|g" \
${TEMPLATE_DIRNAME}/configmaps.template.yaml >
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g"
${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
\
+ ${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g" \
+ ${BUILD_DIRNAME}/configmaps.yaml
cat ${BUILD_DIRNAME}/airflow.yaml
cat ${BUILD_DIRNAME}/configmaps.yaml
# Fix file permissions
+# TODO: Check this - this should be TRAVIS-independent
if [[ "${TRAVIS}" == true ]]; then
sudo chown -R travis.travis $HOME/.kube $HOME/.minikube
fi
-kubectl delete -f $DIRNAME/postgres.yaml
-kubectl delete -f $BUILD_DIRNAME/airflow.yaml
-kubectl delete -f $DIRNAME/secrets.yaml
+kubectl delete -f ${DIRNAME}/postgres.yaml
+kubectl delete -f ${BUILD_DIRNAME}/airflow.yaml
+kubectl delete -f ${DIRNAME}/secrets.yaml
set -e
-kubectl apply -f $DIRNAME/secrets.yaml
-kubectl apply -f $BUILD_DIRNAME/configmaps.yaml
-kubectl apply -f $DIRNAME/postgres.yaml
-kubectl apply -f $DIRNAME/volumes.yaml
-kubectl apply -f $BUILD_DIRNAME/airflow.yaml
+kubectl apply -f ${DIRNAME}/secrets.yaml
+kubectl apply -f ${BUILD_DIRNAME}/configmaps.yaml
+kubectl apply -f ${DIRNAME}/postgres.yaml
+kubectl apply -f ${DIRNAME}/volumes.yaml
+kubectl apply -f ${BUILD_DIRNAME}/airflow.yaml
Review comment:
Yep. I know not everyone will install the pre-commit as an actual pre-commit
hook. But once we have it implemented - we should update .travis.yml to run all
the "pre-commit" hooks on all files in Travis. This would catch the very same
problems but in Travis CI stage not at commit time.
This is exactly what I have done in my POC of pre-commits:
https://travis-ci.org/PolideaInternal/airflow/jobs/552862055 (scroll it down
the log). This job is using the very same pre-commit configuration to run all
the checks on all the relevant files in Airflow (instead of separately running
flake/mypy etc). And it's really fast and uses less overhead for starting
separate machines and preparing the environment (single docker image is built
and used for all tests).
So no matter if pre-commits are *installed* by everyone - but everyone will
have to make sure they pre-commit checks pass in Travis. And you do not need to
install pre-commit checks. You will be able to run them `pre-commit run
shellcheck` for example without installing them as an actual pre-commit hook.
This is the beauty of using pre-commit hook framework. You can (if we decide
to) get it installed as airflow devel dependency (it's from pip) and then you
can start using it right away locally.
I think people will quickly see how useful it is when they will have to fix
those issues and have super easy way to locally reproduce the same checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
