Swalloow commented on a change in pull request #13735:
URL: https://github.com/apache/airflow/pull/13735#discussion_r561920861
##########
File path: chart/templates/workers/worker-deployment.yaml
##########
@@ -214,6 +214,9 @@ spec:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
{{- end }}
+{{- if .Values.workers.extraContainers }}
+{{- toYaml .Values.workers.extraContainers | nindent 8 }}
Review comment:
Not tested with Kubernetes workers.
The `extraVolumes` and `extraVolumeMounts` options also seem to be excluded
from the file.
Need to support it?
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev You can follow `Using additional containers` document in
`README.md`.
An example S3 sync container is as follows:
https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
```
extraContainers:
- name: s3-sync
image: myrepository/s3sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev You can follow `Using additional containers` document in
`README.md`.
An example S3 sync container is as follows:
https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
For version 1.10.14, you need to add it to webserver/scheduler/workers.
```
extraContainers:
- name: s3-sync
image: myrepository/s3sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev You can follow `Using additional containers` document in
`README.md`.
An example S3 sync container is as follows:
https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
When DAG serialization is disabled, you need to add it to
webserver/scheduler/workers.
```
extraContainers:
- name: s3-sync
image: myrepository/s3sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev You can follow `Using additional containers` document in
`README.md`.
An example S3 sync container is as follows:
https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
When DAG serialization is disabled, you need to add it to
webserver/scheduler/workers.
```
extraContainers:
- name: s3-sync
image: myrepository/s3-sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
##########
File path: chart/README.md
##########
@@ -366,6 +366,17 @@ helm install airflow . \
--set data.brokerUrl=redis://redis-user:password@redis-host:6379/0
```
+## Using additional containers
+
+If you are using your own sidecar container, you can add it through the
`extraContainers` value. You can define different containers for scheduler,
webserver and worker pods. For example, a sidecar that syncs DAGs from object
storage.
+
+```bash
Review comment:
sidecar container pattern can be implemented in many ways.
I'm not sure if it is correct to write a specific example here.
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev You can follow `Using additional containers` document in
`README.md`.
example S3 sync container : https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
When DAG serialization is disabled, you need to add it to
webserver/scheduler/workers.
```
extraContainers:
- name: s3-sync
image: myrepository/s3-sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
##########
File path: chart/values.yaml
##########
@@ -391,6 +392,9 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Launch additional containers into scheduler.
+ extraContainers: []
+
# Mount additional volumes into scheduler.
extraVolumes: []
Review comment:
@gardnerdev Here is an example S3 sync container:
https://github.com/Swalloow/s3-sync
If you use EKS, IAM roles for service account settings are required.
When DAG serialization is disabled, you need to add it to
webserver/scheduler/workers.
```
extraContainers:
- name: s3-sync
image: myrepository/s3-sync:latest
imagePullPolicy: Always
volumeMounts:
- name: dags
mountPath: /opt/airflow/dags
env:
- name: AWS_BUCKET
value: airflow-src
- name: KEY_PATH
value: dags
- name: DEST_PATH
value: /opt/airflow/dags
- name: INTERVAL
value: "10"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
