shahar1 commented on code in PR #74913:
URL: https://github.com/apache/airflow/pull/74913#discussion_r4302630762
##########
chart/docs/customizing-workers.rst:
##########
@@ -32,6 +32,50 @@ For example, to set resources on workers for CeleryExecutor:
limits:
cpu: 1
+Separate Celery worker sets
+---------------------------
+
+Use ``workers.celery.sets`` to configure workers for different queues. Each set
+inherits ``workers.celery`` settings and can override them, including
+``extraVolumes`` and ``extraVolumeMounts``. For example, mount different
existing
+PersistentVolumeClaims for two worker sets:
+
+.. code-block:: yaml
+ :caption: values.yaml
+
+ executor: CeleryExecutor
+ workers:
+ celery:
+ enableDefault: false
+ sets:
+ - name: analytics
+ queue: analytics
+ extraVolumes:
+ - name: input-data
+ persistentVolumeClaim:
+ claimName: analytics-input
+ extraVolumeMounts:
+ - name: input-data
+ mountPath: /opt/airflow/input
+ readOnly: true
+ - name: research
+ queue: research
+ extraVolumes:
+ - name: input-data
+ persistentVolumeClaim:
+ claimName: research-input
+ extraVolumeMounts:
+ - name: input-data
+ mountPath: /opt/airflow/input
+ readOnly: true
+
+Create both PersistentVolumeClaims before deploying the chart, and route tasks
+to the corresponding queue. ``enableDefault: false`` disables the default
worker
+set, so this example creates only the ``analytics`` and ``research`` sets.
Review Comment:
**minor** — With `enableDefault: false` nothing consumes the `default`
queue, so any task that is not routed to `analytics` or `research` (including
every task that does not set `queue`) stays queued forever. Users who copy this
example as-is will hit that. Could you either drop `enableDefault: false` from
the example (the two sets then run next to the default workers, which is the
more common setup), or say it explicitly here, e.g.:
```suggestion
set, so this example creates only the ``analytics`` and ``research`` sets.
No worker
then consumes the ``default`` queue, so every task must be routed to one of
these queues.
```
##########
chart/docs/customizing-workers.rst:
##########
@@ -32,6 +32,50 @@ For example, to set resources on workers for CeleryExecutor:
limits:
cpu: 1
+Separate Celery worker sets
+---------------------------
+
+Use ``workers.celery.sets`` to configure workers for different queues. Each set
+inherits ``workers.celery`` settings and can override them, including
+``extraVolumes`` and ``extraVolumeMounts``. For example, mount different
existing
+PersistentVolumeClaims for two worker sets:
+
+.. code-block:: yaml
+ :caption: values.yaml
+
+ executor: CeleryExecutor
+ workers:
+ celery:
+ enableDefault: false
+ sets:
+ - name: analytics
+ queue: analytics
+ extraVolumes:
+ - name: input-data
+ persistentVolumeClaim:
+ claimName: analytics-input
+ extraVolumeMounts:
+ - name: input-data
+ mountPath: /opt/airflow/input
+ readOnly: true
+ - name: research
+ queue: research
+ extraVolumes:
+ - name: input-data
+ persistentVolumeClaim:
+ claimName: research-input
+ extraVolumeMounts:
+ - name: input-data
+ mountPath: /opt/airflow/input
+ readOnly: true
+
+Create both PersistentVolumeClaims before deploying the chart, and route tasks
Review Comment:
**nit** — Each set inherits `replicas` and the HPA/KEDA settings. If a set
runs more than one pod, a `ReadWriteOnce` claim cannot attach on more than one
node, and the extra pods stay in `ContainerCreating`. Maybe worth half a
sentence: "use an access mode such as `ReadOnlyMany` if a set runs more than
one replica".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
