jason810496 opened a new pull request, #59869:
URL: https://github.com/apache/airflow/pull/59869
related: #59747
## Why
It would be really nice to make `breeze k8s dev` as **ONE STEP command** to
setup Airflow on Kubernetes development environment!
Additionally, while testing out the new `breeze k8s dev --dags-path
files/dags --deploy` command locally, I encounter some errors:
1. `ImagePullBackOff` error due to hardcoded image tag as `latest`
```
Generated skaffold config at
/var/folders/r8/sq4khrmx1xnfc7rgqsznh1880000gn/T/skaffold_efl_nwvf/skaffold.yaml
Generating tags...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes ->
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
Checking cache...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Found Locally
Tags used in deployment:
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes ->
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6
Starting deploy...
Loading images into kind cluster nodes...
-
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6
-> Found
Images loaded in 194.644416ms
Helm release airflow not installed. Installing...
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
* timed out waiting for the condition
deploying "airflow": install: exit status 1
```
https://github.com/GoogleContainerTools/skaffold/issues/6804
2. Namespace not found error for Helm
```
Stable repo is already added
Good version of skaffold installed: 2.17.0 in
/Users/jason/Desktop/airflow/.venv/bin
Generating tags...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes ->
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
Checking cache...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Found Locally
Tags used in deployment:
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes ->
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6
Starting deploy...
Loading images into kind cluster nodes...
-
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cdLoaded
Images loaded in 1 minute 7.202 seconds
Helm release airflow not installed. Installing...
Error: INSTALLATION FAILED: create: failed to create: namespaces
"airflow" not found
Cleaning up...
Error: uninstall: Release not loaded: airflow: release: not found
Cleaning up resources encountered an error, will continue to clean up
other resources.
deploying "airflow": install: exit status 1
```
https://github.com/GoogleContainerTools/skaffold/issues/8515
3. It's not an error, but the current `--dags-path` is default as `dags`
related path instead of `files/dags` so we need to explicitly set to
`files/dags` if we need to test out the same dags as `breeze start-airflow`.
```
Stable repo is already added
Good version of skaffold installed: 2.17.0 in
/Users/jason/Desktop/airflow/.venv/bin
Running skaffold without deploying Helm resources. If sync cannot find
pods, rerun with --deploy.
Generating tags...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes ->
ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
Checking cache...
- ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Error
checking cache.
getting hash for artifact
"ghcr.io/apache/airflow/main/prod/python3.10-kubernetes": getting dependencies
for "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes": pattern "dags/**"
did not match any file
```
## What
To fix the above errors:
1. Add `setValueTemplates` skaffold config to resolve image instead of
hardcoding as `latest` image tag to avoid `ImagePullBackOff` error.
2. Add missing `deploy.helm.releases.createNamespace` skaffold config
3. change default value of `--dags-path` from `dags` to `files/dags` (align
as breeze container mount point)
To make `breeze k8s dev` as one step command:
- ask user whether to run `create-cluster` command if found that cluster
hasn't been created.
- if user run `breeze k8s create-cluster` command first, also show the hint
of `breeze k8s dev` so that user don't need to run `breeze k8s
configure-cluster`, `breeze k8s build-k8s-image` ... `breeze k8s
deploy-airflow`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
