potiuk opened a new issue, #44508: URL: https://github.com/apache/airflow/issues/44508
As also experienced by @ashb in https://github.com/apache/airflow/pull/44463, our K8S tests are rather slow to iterate on and fix locally. The way how to run those tests locally is explained in https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst#typical-testing-pattern-for-kubernetes-tests in detail - but there are a fewproblems to solve with them: * The k8s image needs to use PROD base image to be build first and - especially when dependencies change - sometimes, but not always the PROD image needs to be rebuilt with `--rebuild-base-image` flag of `breeze k8s build-k8s-image`. This might cause the Helm chart to not work at all (for example it failed when `asyncpg` was not installed in the base image * Currently when you modify the `./kubernetest_tests` - i.e. tests that you run in the local k8s-dedicated virtulenv environment specially prepared for those tests, they can be re-run right after modification - which is great to iterate on the tests. However, this is NOT the case for "airflow" code that you modify and iterate on. While thisis understandable that if you modify helm chart, you need to redeploy the chart, you also need to rebuild the "k8s image" and re-build and re-upload the image to the kind cluster in order to iterate on your changes. This is SLOW LIKE HELL. Rebuilding the image, re-uploading it and re-deploying chart to reload airflow takes minutes. So in order to fix it we can do several things: * better detection when we need to re-build the image because of dependency change * using uv to install dependencies in PROD image as well (currently it is only used by default in CI image) - this way we could likely even rebuild the image more often as it wil be faster * add extra layer to add missing dependencies (with UV) in the K8S image: https://github.com/apache/airflow/blob/0d6d5f23a2e3861e49b3ea5effeeeb42db4ded94/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L602 - which means that we could continue to only rebuilding the k8s image (without `--rebuild-base-image` flag even if dependencies change) * we could integrate solutions (or build our own) like Telepresence: https://www.telepresence.io/ where local sources of airflow would be synced (via ssh) with the sources in the k8s cluster (for example via local folder mounting or SSH) - this is a bit difficult to implement it to work in all circumstances (Linux/ MacOS/ WSL2) and all kind of docker engines our users might have (Docker Desktop, Orbstack, Podman etc.) - so doing it **righ** is a bit of a challenge. But that should be an ultimate way of iteration where changes to airflow and kuberntes providers could be followed by just pod restart and no need to build an upload new image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
