jscheffl commented on code in PR #61963:
URL: https://github.com/apache/airflow/pull/61963#discussion_r2936589230
##########
chart/values.yaml:
##########
@@ -2473,6 +2473,21 @@ dagProcessor:
# Environment variables to add to dag processor container
env: []
+ # Multiple Dag Processing Configuration
+ # When defined (non-empty), creates a separate Deployment for each entry
instead
+ # of a single dag-processor Deployment. Each entry requires a unique 'name'
and
+ # can specify its own 'replicas' and 'args'. If 'replicas' or 'args' are not
set
+ # for an entry, they fall back to dagProcessor.replicas and
dagProcessor.args.
+ multipleDagProcessing: []
+ # Example:
+ # multipleDagProcessing:
+ # - name: git-dags
+ # replicas: 2
+ # args: ["bash", "-c", "exec airflow dag-processor --subdir
/opt/airflow/dags/git"]
Review Comment:
For me it does not make much sense if the user must specify the full CLI
here. It should rather present "additional args" and not the base shell
command. Or would there be a reason to all another shell with another CLI for
the use case?
##########
chart/values.schema.json:
##########
@@ -4944,6 +4944,38 @@
],
"additionalProperties": false
}
+ },
+ "multipleDagProcessing": {
+ "description": "Multiple Dag Processing configuration.
When non-empty, creates a separate Deployment per entry instead of a single
dag-processor Deployment. Each entry can specify its own replicas and args,
falling back to ``dagProcessor.replicas`` and ``dagProcessor.args`` when not
set.",
+ "type": "array",
+ "default": [],
+ "items": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "name"
+ ],
+ "properties": {
+ "name": {
+ "description": "Unique name for this dag
processor instance. Used as a suffix in the Deployment name.",
+ "type": "string"
+ },
+ "replicas": {
+ "description": "Number of replicas for this
dag processor instance. Falls back to ``dagProcessor.replicas`` if not set.",
+ "type": "integer"
+ },
+ "args": {
+ "description": "Args to use when running this
dag processor instance (templated). Falls back to ``dagProcessor.args`` if not
set.",
Review Comment:
For me it does not make much sense to define multiple Dag processors and if
no args given then falling back to the base. Then you would just have multiple
replicas. If I decide to have multiple deployments then I'd assume there is a
reason for different args == falback does not make sense in my view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
