Miretpl commented on code in PR #58547:
URL: https://github.com/apache/airflow/pull/58547#discussion_r2640789518
##########
chart/newsfragments/58547.significant.rst:
##########
@@ -0,0 +1,24 @@
+Support for Multiple Celery Worker Sets in the Helm Chart
+
+Description
+This PR enhances the Airflow Helm chart to support advanced Celery worker
topologies, enabling more flexible resource allocation and precise autoscaling
configurations.
+
+Why is this needed?
+1. Flexible Worker Topologies
+As Airflow adoption grows, platform teams often need to route tasks
exclusively to specialized worker sets (e.g., GPU-optimized, Memory-optimized)
without maintaining a generic "default" worker.
+
+Enhancement: The new workers.enableDefault flag allows users to configure a
deployment consisting only of specialized worker sets defined in
workers.celery.sets. This provides greater flexibility for teams to design
their worker architecture exactly as needed.
+2. Multi-Queue Autoscaling Support
+Complex workflows often require a single worker set to handle tasks from
multiple specific queues (e.g., queue: "high-priority,vip").
+
+Enhancement: This PR updates the KEDA ScaledObject generation to support
comma-separated queue lists. By using the SQL IN (...) clause, we ensure that
KEDA scales worker sets based on the precise aggregate workload of all their
assigned queues.
+3. Granular Configuration Overrides
+Different worker sets may require different operational strategies within the
same cluster.
+
+Enhancement: This change improves the configuration merge logic, allowing
individual worker sets to override global settings. For example, a user can now
enable KEDA globally but explicitly disable it for a specific worker set that
requires a static number of replicas.
+Changes
+New Feature: Added workers.enableDefault (default: true) to values.yaml.
Review Comment:
```suggestion
New Feature: Added ``workers.enableDefault`` (default: true) to values.yaml.
```
##########
chart/newsfragments/58547.significant.rst:
##########
@@ -0,0 +1,24 @@
+Support for Multiple Celery Worker Sets in the Helm Chart
+
+Description
+This PR enhances the Airflow Helm chart to support advanced Celery worker
topologies, enabling more flexible resource allocation and precise autoscaling
configurations.
+
+Why is this needed?
+1. Flexible Worker Topologies
+As Airflow adoption grows, platform teams often need to route tasks
exclusively to specialized worker sets (e.g., GPU-optimized, Memory-optimized)
without maintaining a generic "default" worker.
+
+Enhancement: The new workers.enableDefault flag allows users to configure a
deployment consisting only of specialized worker sets defined in
workers.celery.sets. This provides greater flexibility for teams to design
their worker architecture exactly as needed.
Review Comment:
```suggestion
Enhancement: The new ``workers.enableDefault`` flag allows users to
configure a deployment consisting only of specialized worker sets defined in
``workers.celery.sets``. This provides greater flexibility for teams to design
their worker architecture exactly as needed.
```
##########
chart/newsfragments/58547.significant.rst:
##########
@@ -0,0 +1,24 @@
+Support for Multiple Celery Worker Sets in the Helm Chart
+
+Description
+This PR enhances the Airflow Helm chart to support advanced Celery worker
topologies, enabling more flexible resource allocation and precise autoscaling
configurations.
+
+Why is this needed?
+1. Flexible Worker Topologies
+As Airflow adoption grows, platform teams often need to route tasks
exclusively to specialized worker sets (e.g., GPU-optimized, Memory-optimized)
without maintaining a generic "default" worker.
+
+Enhancement: The new workers.enableDefault flag allows users to configure a
deployment consisting only of specialized worker sets defined in
workers.celery.sets. This provides greater flexibility for teams to design
their worker architecture exactly as needed.
+2. Multi-Queue Autoscaling Support
+Complex workflows often require a single worker set to handle tasks from
multiple specific queues (e.g., queue: "high-priority,vip").
+
+Enhancement: This PR updates the KEDA ScaledObject generation to support
comma-separated queue lists. By using the SQL IN (...) clause, we ensure that
KEDA scales worker sets based on the precise aggregate workload of all their
assigned queues.
+3. Granular Configuration Overrides
+Different worker sets may require different operational strategies within the
same cluster.
+
+Enhancement: This change improves the configuration merge logic, allowing
individual worker sets to override global settings. For example, a user can now
enable KEDA globally but explicitly disable it for a specific worker set that
requires a static number of replicas.
+Changes
+New Feature: Added workers.enableDefault (default: true) to values.yaml.
+Enhancement: Updated worker-kedaautoscaler.yaml to use SQL IN clause for queue
filtering, supporting multi-queue configurations (e.g., queue: "a,b" -> AND
queue IN ('a','b')).
+Refactor: Standardized template rendering to ensure consistent behavior
between the default worker and workers.celery.sets.
Review Comment:
```suggestion
Refactor: Standardized template rendering to ensure consistent behavior
between the default ``workers`` and ``workers.celery.sets``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
