xEviL commented on pull request #8150: URL: https://github.com/apache/airflow/pull/8150#issuecomment-663355935
@brandonwillard @fengsi @mik-laj I did some experiments yesterday and I was able to define an extra `volume` and `volumeMount` via `pod_template_file` option. What I did is: - run a long running task (`sleep 3600` in a `BashOperator`) in a `KubernetesExecutor` with normal config via `[kubernetes]` section options - dump YAML from k8s pod via `kubectl get pod ... -o yaml` - clean it up (remove obviously "wrong" sections to have: `status`, `metadata.labels`, `command`, `args`) - add my `volume` (from `secret` in my case) and `volumeMount` to the relevant sections - make this `my_worker_template.yaml` file available to Airflow scheduler and webserver (in my case: put it into a `configmap` and mount it into the scheduler pod via its manifest - we are not using the helm chart) - specify the path via `pod_template_file` in `airflow.cfg` - restart the services From what I understood from the source code and the behaviour of this: - YAML from `pod_template_file` is loaded into `PodGenerator` instead of other options and is used as a "pod prototype" - Then it still overrides/initialises varying parameters needed for tasks (like some Airflow config options passed via `env`, also `command` and `args`, and `metadata.labels`) Hope this helps! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
