[jira] [Assigned] (AIRFLOW-3871) Allow Jinja templating recursively on object attributes
[ https://issues.apache.org/jira/browse/AIRFLOW-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Galak reassigned AIRFLOW-3871: -- Assignee: Galak (was: Björn Pollex) > Allow Jinja templating recursively on object attributes > --- > > Key: AIRFLOW-3871 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3871 > Project: Apache Airflow > Issue Type: Wish > Components: operators >Affects Versions: 1.10.0 >Reporter: Galak >Assignee: Galak >Priority: Minor > Fix For: 1.10.6 > > > Some {{Operator}} fields can be templated (using Jinja). Template rendering > only works for string values (either direct values or values stored in > collections). > But a templated string inside a custom class instance won't be rendered > Here is my scenario: > I have a python method {{transform_data_file}} which is designed to call a > command object. This command object constructor > ({{MyAwesomeDataFileTransformer}}) has parameters that could be templated. > These templated parameters are not rendered so far (see > {{BaseOperator.render_template_from_field}} method). > {code} > simple_task = PythonOperator( > task_id='simple_task', > provide_context=True, > python_callable=transform_data_file, > templates_dict={ > 'transformer': MyAwesomeDataFileTransformer( > "/data/{{ dag.dag_id }}/{{ ts }}/input_file", > "/data/{{ dag.dag_id }}/{{ ts }}/output_file", > ) > }, > dag=dag > ) > {code} > I have 3 alternatives in mind to allow rendering inner attributes: > # Either define an Abstract Base Class declaring an abstract method > {{render_template}}; then my command object would have to extend this > Abstract Base Class, and then implement {{render_template}} method. > # Or use duck typing in {{BaseOperator.render_template_from_field}} to call > {{render_template}} method when it exists on templated custom objects; then > my command object would just have to implement {{render_template}} method. > # Or traverse object attributes when rendering templates and call > {{BaseOperator.render_template}} recursively; then my command object would > not need any change > My preferred solution is the 3rd one, but I would like to hear about your > opinion on this before. Maybe is there a 4th and better solution? > I would be glad to submit a PR if this functionality is accepted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-3871) Allow Jinja templating recursively on object attributes
[ https://issues.apache.org/jira/browse/AIRFLOW-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Björn Pollex reassigned AIRFLOW-3871: - Assignee: Björn Pollex > Allow Jinja templating recursively on object attributes > --- > > Key: AIRFLOW-3871 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3871 > Project: Apache Airflow > Issue Type: Wish > Components: operators >Affects Versions: 1.10.0 >Reporter: Galak >Assignee: Björn Pollex >Priority: Minor > > Some {{Operator}} fields can be templated (using Jinja). Template rendering > only works for string values (either direct values or values stored in > collections). > But a templated string inside a custom class instance won't be rendered > Here is my scenario: > I have a python method {{transform_data_file}} which is designed to call a > command object. This command object constructor > ({{MyAwesomeDataFileTransformer}}) has parameters that could be templated. > These templated parameters are not rendered so far (see > {{BaseOperator.render_template_from_field}} method). > {code} > simple_task = PythonOperator( > task_id='simple_task', > provide_context=True, > python_callable=transform_data_file, > templates_dict={ > 'transformer': MyAwesomeDataFileTransformer( > "/data/{{ dag.dag_id }}/{{ ts }}/input_file", > "/data/{{ dag.dag_id }}/{{ ts }}/output_file", > ) > }, > dag=dag > ) > {code} > I have 3 alternatives in mind to allow rendering inner attributes: > # Either define an Abstract Base Class declaring an abstract method > {{render_template}}; then my command object would have to extend this > Abstract Base Class, and then implement {{render_template}} method. > # Or use duck typing in {{BaseOperator.render_template_from_field}} to call > {{render_template}} method when it exists on templated custom objects; then > my command object would just have to implement {{render_template}} method. > # Or traverse object attributes when rendering templates and call > {{BaseOperator.render_template}} recursively; then my command object would > not need any change > My preferred solution is the 3rd one, but I would like to hear about your > opinion on this before. Maybe is there a 4th and better solution? > I would be glad to submit a PR if this functionality is accepted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)