Galak created AIRFLOW-3871:
------------------------------
Summary: Allow Jinja templating recursively on object attributes
Key: AIRFLOW-3871
URL: https://issues.apache.org/jira/browse/AIRFLOW-3871
Project: Apache Airflow
Issue Type: Wish
Components: operators
Affects Versions: 1.10.0
Reporter: Galak
Some {{Operator}} fields can be templated (using Jinja). Template rendering
only works for string values (either direct values or values stored in
collections).
But a templated string inside a custom class instance won't be rendered
Here is my scenario:
I have a python method {{transform_data_file}} which is designed to call a
command object. This command object constructor
({{MyAwesomeDataFileTransformer}}) has parameters that could be templated.
These templated parameters are not rendered so far (see
{{BaseOperator.render_template_from_field}} method).
{code}
simple_task = PythonOperator(
task_id='simple_task',
provide_context=True,
python_callable=transform_data_file,
templates_dict={
'transformer': MyAwesomeDataFileTransformer(
"/data/{{ dag.dag_id }}/{{ ts }}/input_file",
"/data/{{ dag.dag_id }}/{{ ts }}/output_file",
)
},
dag=dag
)
{code}
I have 3 alternatives in mind to allow rendering inner attributes:
# Either define an Abstract Base Class declaring an abstract method
{{render_template}}; then my command object would have to extend this Abstract
Base Class, and then implement {{render_template}} method.
# Or use duck typing in {{BaseOperator.render_template_from_field}} to call
{{render_template}} method when it exists on templated custom objects; then my
command object would just have to implement {{render_template}} method.
# Or traverse object attributes when rendering templates and call
{{BaseOperator.render_template}} recursively; then my command object would not
need any change
My preferred solution is the 3rd one, but I would like to hear about your
opinion on this before. Maybe is there a 4th and better solution?
I would be glad to submit a PR if this functionality is accepted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
