potiuk commented on issue #6496: [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing. Depends on [AIRFLOW-5827] [AIRFLOW-5830] [AIRFLOW-5829] URL: https://github.com/apache/airflow/pull/6496#issuecomment-549293886 Notes especially to @dimberman and @gerardo (but also @ashb and @mik-laj you might be interested). As preparation work for the production images I improved the way how the kind cluster is run. First of all I realised that the current cluster does not test really the latest sources - but instead whatever could be pulled from the latest master CI image on Dockerhub (which was usually at least one commit old). I think @gerardo mentioned it last time we spoke. Luckily we did not have breaking changes :) This change fixes it - I am now building the Kubernetes image from locally built image in the host and run `docker save` it to a tar archive which is then mounted to airflow-testing container so that it can be loaded to DinD docker and further loaded to the KinD cluster. The .tar image contains latest sources: * In CI it is done right after the image is built (so has latest sources by definition). * In Breeze I ask the user to "force build" the image always when the user enters Breeze with `--start-kubernetes-cluster` flag. This way the image with latest sources will be build/saved and made available to airflow-testing container via mounted volume. @dimberman - you migth also be interested that I made the development workflow for Kubernetes tests with Breeze as nice as possible (we will later improve it - maybe with the production image). The original "Kind" change was working fine for CI (except not taking latest sources), but running it for local development was not really feasible. You really needed to rebuild everything from the scratch when you entered the environment (which took like 10-20 minutes on my PC) and there was no easy way to test the latest sources - short of forwarding, building and pushing your own images to dind container :(. I optimised it now so that when you have not closed the Breeze environment it will take far less time: * kind cluster is not deleted every time - it is reused when you exit/enter Breeze (and do not stop environment) * if you enter the cluster with --start-kubernetes-cluster, the CI image will be rebuilt taking latest sources, then kubernetes image is build out of it and saved/mounted to airflow-testing container * If you do not stop the environment, the DiND container is running so it keeps both Kind cluster running as well as the kubernetes images pushed. * this means that if we rebuild the image with latest sources, only few last layers change and it is much faster to load the image - both to DiND and to Cluster. * I've done it in the way that if you have enough power/memory, you could run multiple clusters (each for different kubernetes version) and multiple airflows - one for each python version, all in one DiND. That might be useful when we migrate out of Travis. * still if you stop the environment everything will be deleted and whole setup will be setup from the scratch (which is good for cleanup etc.) * It should be easy to switch to an external cluster with the current setup in the future. * I added instructions how to run the Kubernetes tests in Breeze It's not ideal and rather slow, but at least it is automated now and you can run Kubernetes tests with your latest sources rather easily/automatically.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
