potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] URL: https://github.com/apache/airflow/pull/4938#discussion_r294139989
########## File path: scripts/ci/ci_before_install.sh ########## @@ -0,0 +1,34 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -xeuo pipefail + +MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +export KUBERNETES_VERSION=${KUBERNETES_VERSION:=} +# Required for K8s v1.10.x. See +# https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783 +if [[ ! -z "${KUBERNETES_VERSION}" ]]; then + sudo mount --make-shared / + sudo service docker restart +fi +pip install --upgrade pip Review comment: Hey @Fokko @ashb - I fixed all the problems except the kubernetes tests (hopefully @gerardo might help with that). Everything else is green :) and nice: https://travis-ci.org/apache/airflow/builds/546571775 . Please take another look. In this latest version there are a few things improved: - I found a way how to speed up the builds even if setup.py changes. Please take a look at Dockerfile modifications but basically I am pre-installing the dependencies from setup.py downloaded from Airflow master of github repo. This way the dependencies are already there when setup.py is added so changing setup.py will not invalidate the cache and only new/changed dependencies will be installed - that is few minutes saved when dependencies change. Still we will have the clean-cache build running daily with all caching disabled - this way we will detect any "fresh/full" installation problems, so there is no risk. - All the pylint/docs/mypy/flake are now run inside Docker :). And they are fully reproducible locally as well without having to install anything (running ci_pylint.sh/ci_docs.sh etc. locally will download/rebuild the right docker image and run the same checks that are run on CI. No more diifferent results locally and in CI! @BasPH -that might make our pylint experience much better. I also updated CONTRIBUTING.md to explain how to run those. - Also @BasPH - I've added Pylint section to the CONTRIBUTING.md explaining the process we follow for pylint checks. I hope it might be useful. Please take a look and hopefully Kubernetes will be solved within days and we can merge it finally. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
