[jira] [Assigned] (AIRFLOW-3871) Allow Jinja templating recursively on object attributes

2019-09-18 Thread Galak (Jira)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Galak reassigned AIRFLOW-3871:
--

Assignee: Galak  (was: Björn Pollex)

> Allow Jinja templating recursively on object attributes
> ---
>
> Key: AIRFLOW-3871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3871
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: operators
>Affects Versions: 1.10.0
>Reporter: Galak
>Assignee: Galak
>Priority: Minor
> Fix For: 1.10.6
>
>
> Some {{Operator}} fields can be templated (using Jinja). Template rendering 
> only works for string values (either direct values or values stored in 
> collections).
> But a templated string inside a custom class instance won't be rendered
> Here is my scenario: 
> I have a python method {{transform_data_file}} which is designed to call a 
> command object. This command object constructor 
> ({{MyAwesomeDataFileTransformer}}) has parameters that could be templated. 
> These templated parameters are not rendered so far (see 
> {{BaseOperator.render_template_from_field}} method). 
> {code}
> simple_task = PythonOperator(
> task_id='simple_task',
> provide_context=True,
> python_callable=transform_data_file,
> templates_dict={
>   'transformer': MyAwesomeDataFileTransformer(
> "/data/{{ dag.dag_id }}/{{ ts }}/input_file",
> "/data/{{ dag.dag_id }}/{{ ts }}/output_file",
> )
> },
> dag=dag
> )
> {code}
> I have 3 alternatives in mind to allow rendering inner attributes:
> # Either define an Abstract Base Class declaring an abstract method 
> {{render_template}}; then my command object would have to extend this 
> Abstract Base Class, and then implement {{render_template}} method.
> # Or use duck typing in {{BaseOperator.render_template_from_field}} to call 
> {{render_template}} method when it exists on templated custom objects; then 
> my command object would just have to implement {{render_template}} method.
> # Or traverse object attributes when rendering templates and call 
> {{BaseOperator.render_template}} recursively; then my command object would 
> not need any change
> My preferred solution is the 3rd one, but I would like to hear about your 
> opinion on this before. Maybe is there a 4th and better solution?
> I would be glad to submit a PR if this functionality is accepted.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (AIRFLOW-3871) Allow Jinja templating recursively on object attributes

2019-03-05 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Björn Pollex reassigned AIRFLOW-3871:
-

Assignee: Björn Pollex

> Allow Jinja templating recursively on object attributes
> ---
>
> Key: AIRFLOW-3871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3871
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: operators
>Affects Versions: 1.10.0
>Reporter: Galak
>Assignee: Björn Pollex
>Priority: Minor
>
> Some {{Operator}} fields can be templated (using Jinja). Template rendering 
> only works for string values (either direct values or values stored in 
> collections).
> But a templated string inside a custom class instance won't be rendered
> Here is my scenario: 
> I have a python method {{transform_data_file}} which is designed to call a 
> command object. This command object constructor 
> ({{MyAwesomeDataFileTransformer}}) has parameters that could be templated. 
> These templated parameters are not rendered so far (see 
> {{BaseOperator.render_template_from_field}} method). 
> {code}
> simple_task = PythonOperator(
> task_id='simple_task',
> provide_context=True,
> python_callable=transform_data_file,
> templates_dict={
>   'transformer': MyAwesomeDataFileTransformer(
> "/data/{{ dag.dag_id }}/{{ ts }}/input_file",
> "/data/{{ dag.dag_id }}/{{ ts }}/output_file",
> )
> },
> dag=dag
> )
> {code}
> I have 3 alternatives in mind to allow rendering inner attributes:
> # Either define an Abstract Base Class declaring an abstract method 
> {{render_template}}; then my command object would have to extend this 
> Abstract Base Class, and then implement {{render_template}} method.
> # Or use duck typing in {{BaseOperator.render_template_from_field}} to call 
> {{render_template}} method when it exists on templated custom objects; then 
> my command object would just have to implement {{render_template}} method.
> # Or traverse object attributes when rendering templates and call 
> {{BaseOperator.render_template}} recursively; then my command object would 
> not need any change
> My preferred solution is the 3rd one, but I would like to hear about your 
> opinion on this before. Maybe is there a 4th and better solution?
> I would be glad to submit a PR if this functionality is accepted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)