Mariusz Górski created AIRFLOW-5774:
---------------------------------------
Summary: PapermillOperator fails to render parameters
Key: AIRFLOW-5774
URL: https://issues.apache.org/jira/browse/AIRFLOW-5774
Project: Apache Airflow
Issue Type: Bug
Components: utils
Affects Versions: 1.10.6
Reporter: Mariusz Górski
Assignee: Mariusz Górski
*Problem*
When running sample PapermillOperator described in docs, it fails with
{code:java}
//[2019-10-27 10:48:53,134] {{taskinstance.py:1047}} ERROR - Can't compile non
template nodes
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 922, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.7/site-packages/airflow/operators/papermill_operator.py",
line 71, in execute
logging.info(inlet.as_dict())
File "/usr/local/lib/python3.7/site-packages/airflow/lineage/datasets.py",
line 85, in as_dict
attributes[key] = env.from_string(value).render(**self.context)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line
880, in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line
581, in compile
defer_init=defer_init)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line
543, in _generate
optimized=self.optimized)
File "/usr/local/lib/python3.7/site-packages/jinja2/compiler.py", line 78, in
generate
raise TypeError('Can\'t compile non template nodes')
TypeError: Can't compile non template nodes
{code}
*Reason*
It's because: `class NoteBook(DataSet)` expects `parameters` to be `strings`,
while papermill `execute_notebook` function requires `parameters` to be
`dicts`. When we pass dict to NoteBook.parameters, then jinja templating fails
when dict is passed to `env.from_string(value).render(**context)` call.
*Solution:*
Handling of nested structs in DataSet class
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
