Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
shahar1 closed issue #50206: Allow templating for more arguments in RunPipelineJobOperator URL: https://github.com/apache/airflow/issues/50206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
ferenc-hechler commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2867545483 There is already a PR for this: #50220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
gyli commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2867058537 Hi @ferenc-hechler shall I assign this issue to you if you want to create a PR for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
MaksYermak commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2866156951 > I don't see a problem adding these fields to template fields. [@MaksYermak](https://github.com/MaksYermak) do you remember why they were not template field at the beginning? At the beginning we, always, don't know what fields users want to have as templated. We add only the most useful to `template_fields` and then extended it. I, also, do not see any problems with these new fields. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
gyli commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2856873755 I don't see a problem adding these fields to template fields. @MaksYermak do you remember why they were not template field at the beginning? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
ferenc-hechler commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2851988853 There exists also a discussion for this: #48627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
boring-cyborg[bot] commented on issue #50206: URL: https://github.com/apache/airflow/issues/50206#issuecomment-2851229651 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] Allow templating for more arguments in RunPipelineJobOperator [airflow]
ferenc-hechler opened a new issue, #50206:
URL: https://github.com/apache/airflow/issues/50206
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
2.10.2
### Operating System
GCP Composer 2.9.11
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### What happened
I tried to use jinja templating to set parameters like "template_path",
"pipeline_root" or "service_account".
The templates are not replaced with actual values, instead of
service_account is set to the string `{{ var.value... }}`.
### What you think should happen instead
templates should also be supported for these fields and the value of the
environment variable should be given.
### How to reproduce
Executing the following task fails:
```
run_task = RunPipelineJobOperator(
task_id="run_pipeline_job",
display_name="Run Pipeline Job",
template_path="{{ var.value.KFP_TEMPLATE_URL }}",
parameter_values={},
region="europe-west3",
project_id="my-project-id",
)
```
The AirFlow variable `KFP_TEMPLATE_URL` is set to a valid template uploaded
to an Kubeflow Pipeline Artifact Registry.
In the error message it is to see, that template_path was not resolved.
invalid URL "{{ var.value.KFP_TEMPLATE_URL }}"
### Anything else
I was able to activate templating by inheriting the RunPipelineJobOperator
and extend the class variable tempalte_fields:
https://github.com/apache/airflow/blob/e9c49d6056e60b41d682aa9a4f1b7eba08e6aefd/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L110-L115
```
template_fields = [
"region",
"project_id",
"input_artifacts",
"impersonation_chain",
[+] "template_path",
[+] "pipeline_root",
[+] "parameter_values",
[+] "service_account",
]
```
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
