potiuk commented on pull request #8265: URL: https://github.com/apache/airflow/pull/8265#issuecomment-638250690
Hey @dimberman @schnie @kaxil @ashb @turbaszek @mik-laj -> I think this one is pretty much ready for the final round of reviews. I think it either succeed with tests or is very close to succeed. It should also be fairly optimised in terms of build/test speed - using mostly caching where possible. It should also help a lot with further iterations on the production image and Helm chart. The changes introduced: 1) The scripts for Kubernetes Setup are moved out from the CI container. They are all run in host and running Kubenetes tests does not require CI image at all. It is neither built nor used during the tests. 2) We have scripts and breeze commands to manage KinD cluster used for testing. It's done in a portable way - I download Kind and Kubectl as needed automatically (in the right version) to .build directory on the host when you run the commands so it should be really easy for everone to reproduce the tests. You have simple `kind-cluster stop/start/status/restart/deploy/test` commands in breeze so that you can easily setup your machine for testing with Kubernetes Cluster. 3) We have a separate `deploy` command that builds production image (if needed) and prepares the *-kubernetes image that is then sent to KinD and use as "airflow image" during tests. This step is automated and when we go to Helm Chart - this is pretty much the only part that will have to be replaced. Simply the "deploy" function should be written in the way that it should send the image to Kind, start Airflow on the Kind Cluster and wait for the webserver to be ready. We have "ci_deploy_app_to_kuberenetes.sh" script that utilises that function on top of Breeze command. Deployment is now much faster than before (1-2 minutes) so iteration speed is much more reasonable. 4) The Kubernetes tests (those that require real kubernetes) are moved to separate `kubernetes_tests` folder - this way it's clear they are only run during Kubernetes tests. You can run it on your own if you have airflow virtualenv but for the sake of repeatabiity and ease of use there is also a "ci_run_kubernetes_tests.sh" that will setup approproate virtualenv automatically and run the tests individually` but alsso there is `breeze kind-cluster-test` command that should execute the tests. But you can run the tests directly from the IDE as well if you pick the right virtualenv. Those tests are completely separated from the CI image. 5) I also added some extra logging info and refactored slightly KubernetesExecutor tests and made them a bit more robust. For example they were written in the way that if you manually run any DAG on the cluster - once - then this cluster became unusable for testing (tests were failing without any meaningful explanation) :(. Those tests are still far from robust but now at least you can see what's going on and they are slightly more resilient. It's quite a change but it's been already reviewed several times - I hope to merge it quickly - this will complete our GA migration and we will be able to iterate much faster on both prod image and chart and merge those two eventually. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
