potiuk commented on PR #25678: URL: https://github.com/apache/airflow/pull/25678#issuecomment-1233360504
I believe this one should be already green and mergable, and it is what I wanted it to be. It's WAY nicer comparing to what we had before and as explained in #26091 we will be able to further improve the development experience of Airlfow running on K8S for everyone as a follow-up. @dstandish @ephraimbuddy @jedcunningham - I think you were one of the users there (or maybe non-users because of not-so-great experience). This Python-Breeze version has a lot of improvements comparing to the legacy bash one: * The CI has 1-1 mapping to the local development commands. You follow exactly the same sequence locally to run tests locally as you do in CI (screenshot below) * The commands are consistent with the rest of the commands of breeze , and allow both local single-cluster running as well as CI running in parallel * The documentation is simplified - some of the duplication was removed and it's much clearer now how to approach it IMHO * Also the commands to setup testable Airlfow instance are pretty much "wizard-like" (@rossturk that is the result of our discussions). When you create cluster, it tells you "NEXT STEP:" that you should configure it and tells you the command, when you configure it it asks you to build k8s as "NEXT STEP" and tells the command, and so on. It basically guides the user step-by-step on what to do next. You can literally re-run all k8c tests by running `breeze k8s create-cluster` and following the "NEXT STEPS" instructions. * The "K8S upgrade" step is now part of the regular jobs. We save a lot of time for setting up the clusters and deploying airlfow again as they are already set-up in previous steps and we simply need to add one "upgrade" step with some changed valiues (@ephraimbuddy - you will be happy). This makes the tests less resource hungry and faster. * I found and fixed a few stability issues that plagued us with the Helm tests. They should be much more solid now in general. That included necessary increase in our CI machine - to add more inotify watchers on system level for example (and few other lower-level stuff). * It's all Python * The output of CI is really nice and tells exactly what happens Example succesful run with multiple clusters in parallel: https://github.com/apache/airflow/runs/8114807752?check_suite_focus=true Some screenshots: Steps in the CI job:  Job summary:  Job details after unfolding:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
