Miretpl commented on code in PR #61039:
URL: https://github.com/apache/airflow/pull/61039#discussion_r3215450097
##########
chart/values.yaml:
##########
@@ -2309,6 +2309,39 @@ dagProcessor:
# classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
# kwargs: {}
+ # Per-bundle deployment option
+ # When enabled, creates a separate deployment for each bundle in
`dagBundleConfigList`
+ deployPerBundle:
Review Comment:
I think that it would be nice to have a possibility to deploy a couple of
dag bundles within one dag processor and deploy one dag bundle per dag
processor at the same time. Looking at the workers sets after removal of
deprecations (WIP PR #66671), the generation is much faster than it was, and
also, I think that there is some room for further improvements. If we agree
that the Worker Set after deprecation removal is good enough in terms of:
1. speed
2. maintenance
3. readability
perspectives, I think we could go with a similar structure for the
`dagProcessor` section with the addition of e.g.:
```yaml
multipleDagProcessors: []
```
section which would work the same way as `workers.celery.sets` section
(basically possible overwrite of any field defined under `dagProcessor`
section). We already have in `dagProcessor.dagBundleConfigList` field for
specifying the bundles, so the users would be able to specify many different
bundles per given dag processor deployments.
##########
chart/values.yaml:
##########
@@ -2309,6 +2309,39 @@ dagProcessor:
# classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
# kwargs: {}
+ # Per-bundle deployment option
+ # When enabled, creates a separate deployment for each bundle in
`dagBundleConfigList`
+ deployPerBundle:
Review Comment:
I think that it would be nice to have a possibility to deploy a couple of
dag bundles within one dag processor and deploy one dag bundle per dag
processor at the same time. Looking at the workers sets after removal of
deprecations (WIP PR #66671), the generation is much faster than it was, and
also, I think that there is some room for further improvements. If we agree
that the Worker Set after deprecation removal is good enough in terms of:
1. speed
2. maintenance
3. readability
perspectives, I think we could go with a similar structure for the
`dagProcessor` section with the addition of e.g.:
```yaml
multipleDagProcessors: []
```
section which would work the same way as `workers.celery.sets` section
(basically possible overwrite of any field defined under `dagProcessor`
section). We already have in `dagProcessor.dagBundleConfigList` field for
specifying the bundles, so the users would be able to specify many different
bundles per given dag processor deployments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
