[
https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164156#comment-17164156
]
ASF GitHub Bot commented on AIRFLOW-3126:
-----------------------------------------
xEviL commented on pull request #8150:
URL: https://github.com/apache/airflow/pull/8150#issuecomment-663355935
@brandonwillard @fengsi @mik-laj
I did some experiments yesterday and I was able to define an extra `volume`
and `volumeMount` via `pod_template_file` option. What I did is:
- run a long running task (`sleep 3600` in a `BashOperator`) in a
`KubernetesExecutor` with normal config via `[kubernetes]` section options
- dump YAML from k8s pod via `kubectl get pod ... -o yaml`
- clean it up (remove obviously "wrong" sections to have: `status`,
`metadata.labels`, `command`, `args`)
- add my `volume` (from `secret` in my case) and `volumeMount` to the
relevant sections
- make this `my_worker_template.yaml` file available to Airflow scheduler
and webserver (in my case: put it into a `configmap` and mount it into the
scheduler pod via its manifest - we are not using the helm chart)
- specify the path via `pod_template_file` in `airflow.cfg`
- restart the services
From what I understood from the source code and the behaviour of this:
- YAML from `pod_template_file` is loaded into `PodGenerator` instead of
other options and is used as a "pod prototype"
- Then it still overrides/initialises varying parameters needed for tasks
(like some Airflow config options passed via `env`, also `command` and `args`,
and `metadata.labels`)
Hope this helps!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> kubernetes executor not allowing new PV
> ---------------------------------------
>
> Key: AIRFLOW-3126
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3126
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
> Affects Versions: 1.10.0
> Reporter: Rahul Singh
> Assignee: Brandon Willard
> Priority: Major
>
> Kubernetes executor allows only two PV one for dag and other for logs , in
> project scenario there are multiple PV needed , like separate PV to store
> data , PV to store common script files etc . Current implementation is very
> static to only two PV.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
