potiuk commented on pull request #12163: URL: https://github.com/apache/airflow/pull/12163#issuecomment-723658950
Hey @kaxil @dimberman @mik-laj @ash (others) -> I really hope that the (currently failing in master) kubernetes test on master will be fixed with this change. I could not reproduce the failure locally, so I figured out that it is likely the "environment" that causes the problem rather than the test. I've just repeated everything from the scratch and I got a repeatable, stable way of running all tests (and they all succeed locally). On my way of fixing the "environment" I made this PR and it introduces several improvements. I really hope all of them combined will bring back sanity aand tests will turn green :). I can later split this into separate PRs, if you think it makes sense - let me know what you think please Here is a summary of the changes: * introducing k9s as first-class-citizen in Kubernetes tests. It's GREAT tool that is so helpful when debugging! * I've replaced kubectl port-forward which much more robust solution - combining externalPortMapping and custom NodePort resource deployed for testing. This should be super-robust now - because there is no external kubectl command running in the background potentially blocking the 8080 port number for listening - we had do implement some workaround for that before but it was rather brittle and could be a reason for the test failing. Now it is super-straightforward because it is built in Kind - and stopping Kind, stops it immediately as well. This allowed to vastly simplify the scripts that setup kind on CI. No more need to try different port numbers. * I separated out KUBECONFIG for testing the Airflow tests - so that the kind cluster setup does not interfers/override your local .kube/config. This makes it totally independent from any other k8s work you might be doing * Last-but-not-least : I added even more detailed description on how to test kubernetes tests (including screenshots on how to configure and debug the kubernetes tests using IntelliJ and teh virtualenv created by Breeze). I have a feeling that people are afraid doing this now, because they do not know how to do it - but with k9s, KUBECONFIG change and port-forwarding built-in Kind and the screenshots, I hope it will be easier to solve/test such problems (and add more tests as we wanted to do for a long time) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
