dimberman commented on a change in pull request #5621: [AIRFLOW-4999] Pull and build local work on both images URL: https://github.com/apache/airflow/pull/5621#discussion_r305926211
########## File path: CONTRIBUTING.md ########## @@ -424,72 +440,84 @@ KUBERNETES_VERSION==v1.13.0 BACKEND=postgres ENV=kubernetes ./scripts/ci/local_c * PYTHON_VERSION might be one of 3.5/3.6 * BACKEND might be one of postgres/sqlite/mysql -* ENV might be one of docker/kubernetes +* ENV might be one of docker/kubernetes/bare * KUBERNETES_VERSION - required for Kubernetes tests - currently KUBERNETES_VERSION=v1.13.0. -The kubernetes env might not work locally as easily as other tests because it requires your host -to be setup properly (specifically it installs minikube cluster locally on your host and depending +The following environments are possible: + * The "docker" environment (default) - it starts all dependencies required by full suite of tests + including postgres/mysql and other components. It's quite resource heavy so do not forget to stop + the environment using [Stop environment](#stopping-the-environment) when you finish working wit it. It + can also slow down your local machine. + * The "kubernetes" environment is used to run tests with kubernetes cluster. + * The "bare" environment is used to run a bare docker instance of Airflow without any external dependencies + which means that you can only run it with sqlite backend. + +Note! The kubernetes env might not work locally as easily as other tests because it requires your host +to be setup properly (specifically it installs minikube cluster locally on your host and depending on your machine setting it might or might not work out of the box. We are working on making the kubernetes tests more easily reproducible locally in the future. ### Stopping the environment Docker-compose environment starts a number of docker containers and keep them running. -You can tear them down by running +You can tear them down by running [/scripts/ci/local_ci_stop_environment.sh](scripts/ci/local_ci_stop_environment.sh) ### Fixing file/directory ownership On Linux there is a problem with propagating ownership of created files (known Docker problem). Basically -files and directories created in container are not owned by the host user (but by the root user in our case). -This might prevent you from switching branches for example if files owned by root user are created within -your sources. In case you are on Linux host and haa some files in your sources created by the root user, -you can fix the ownership of those files by running +files and directories created in container are not owned by the host user (but by the root user in our case). +This might prevent you from switching branches for example if files owned by root user are created within +your sources. In case you are on Linux host and haa some files in your sources created by the root user, +you can fix the ownership of those files by running [scripts/ci/local_ci_fix_ownership.sh](scripts/ci/local_ci_fix_ownership.sh) script. ### Building the images -You can manually trigger building of the local CI image using -[scripts/ci/local_ci_build.sh](scripts/ci/local_ci_build.sh). +You can manually trigger building of the local images using +[scripts/ci/local_ci_build.sh](scripts/ci/local_ci_build.sh). ### Force pulling the images -You can later force-pull the images before building it locally so that you are sure that you download +You can also force-pull the images before building them locally so that you are sure that you download latest images from DockerHub repository before building. This can be done with [scripts/ci/local_ci_pull_and_build.sh](scripts/ci/local_ci_pull_and_build.sh) script. ## Cleaning up cached Docker images/containers Note that you might need to cleanup your Docker environment occasionally. The images are quite big (1.5GB for both images needed for static code analysis and CI tests). And if you often rebuild/update -images you might end up with some unused image data. +images you might end up with some unused image data. + +Cleanup can be performed with `docker system prune` command. + +In case you have huge problems with disk space and want to clean-up all image data you can run +`docker system prune --all`. You might need to [Stop the environment](#stopping-the-environment) before +in order to clean everything including running container or even restart the whole docker engine. -Cleanup can be performed with `docker system prune` command. In case you have huge problems with disk space -and want to clean-up all image data you can run `docker system prune --all`. You might need to -[Stop the environment](#stopping-the-environment) in order to clean everything including running containers. +You can check if your docker is clean by running `docker images --all` and `docker ps --all` - both +should return an empty list of images and containers respectively. If you are on Mac OS and you end up with not enough disk space for Docker you should increase disk space available for Docker. See [Docker for Mac - Space](https://docs.docker.com/docker-for-mac/space/) for details. ## Troubleshooting -In case you have problems with the Docker Compose environment - try the following: +In case you have problems with the Docker Compose environment - try the following (after each step you Review comment: If you are having problems with the Docker Compose environment ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
