[ 
https://issues.apache.org/jira/browse/AIRFLOW-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shire updated AIRFLOW-573:
----------------------------------
    Description: 
A dot/period in a templated string appears to cause jinja template errors in 
Airflow.

example:

I constructed a dictionary to pass as an environment into BashOperator
i.e.
envvars = {'use_this_script':'/Users/mshire/x.sh'}
task4 = BashOperator(
    bash_command="env",
    task_id="test_env_variables",
    env=envvars,
    dag=dag
    )

If I remove the dot/period (i.e. ".") in "x.sh" then it works.  Otherwise I get 
an error below.  I have tried it directly in python with jinja2 and it works, 
but not in Airflow.

> airflow test scratch test_env_variables 2016-10-01
[2016-10-16 00:39:32,981] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-10-16 00:39:33,374] {models.py:154} INFO - Filling up the DagBag from 
/Users/mshire/airflow/dags
envvars= {'use_this_script': '/Users/mshire/x.sh'}
[2016-10-16 00:39:33,479] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2016-10-16 00:39:33,480] {models.py:1219} INFO - Executing 
<Task(BashOperator): test_env_variables> on 2016-10-01 00:00:00
[2016-10-16 00:39:33,489] {models.py:1286} ERROR - /Users/mshire/x.sh
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1233, 
in run
    self.render_templates()
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1409, 
in render_templates
    rendered_content = rt(attr, content, jinja_context)
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2019, 
in render_template
    return self.render_template_from_field(attr, content, context, jinja_env)
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
in render_template_from_field
    for k, v in list(content.items())}
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
in <dictcomp>
    for k, v in list(content.items())}
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2017, 
in render_template
    return jinja_env.get_template(content).render(**context)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
774, in _load_template
    cache_key = self.loader.get_source(self, name)[1]
  File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in 
get_source
    raise TemplateNotFound(template)
TemplateNotFound: /Users/mshire/x.sh
[2016-10-16 00:39:33,490] {models.py:1306} INFO - Marking task as FAILED.

  was:
A dot/period in a templated string appears to cause jinja template errors in 
Airflow.

example:

I constructed a dictionary to pass as an environment into BashOperator
i.e.
envvars = {'use_this_script':'/Users/mshire/x.sh'}
print "env=",env
task4 = BashOperator(
    bash_command="env",
    task_id="test_env_variables",
    env=envvars,
    dag=dag
    )

If I remove the dot/period (i.e. ".") in "x.sh" then it works.  Otherwise I get 
an error below.  I have tried it directly in python with jinja2 and it works, 
but not in Airflow.

> airflow test scratch test_env_variables 2016-10-01
[2016-10-16 00:39:32,981] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-10-16 00:39:33,374] {models.py:154} INFO - Filling up the DagBag from 
/Users/mshire/airflow/dags
env= {'use_this_script': '/Users/mshire/x.sh'}
[2016-10-16 00:39:33,479] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2016-10-16 00:39:33,480] {models.py:1219} INFO - Executing 
<Task(BashOperator): test_env_variables> on 2016-10-01 00:00:00
[2016-10-16 00:39:33,489] {models.py:1286} ERROR - /Users/mshire/x.sh
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1233, 
in run
    self.render_templates()
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1409, 
in render_templates
    rendered_content = rt(attr, content, jinja_context)
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2019, 
in render_template
    return self.render_template_from_field(attr, content, context, jinja_env)
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
in render_template_from_field
    for k, v in list(content.items())}
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
in <dictcomp>
    for k, v in list(content.items())}
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2017, 
in render_template
    return jinja_env.get_template(content).render(**context)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
774, in _load_template
    cache_key = self.loader.get_source(self, name)[1]
  File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in 
get_source
    raise TemplateNotFound(template)
TemplateNotFound: /Users/mshire/x.sh
[2016-10-16 00:39:33,490] {models.py:1306} INFO - Marking task as FAILED.


> Jinja2 template not rendering if the string has a period/dot in it
> ------------------------------------------------------------------
>
>                 Key: AIRFLOW-573
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-573
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: Airflow 2.0
>            Reporter: Michael Shire
>
> A dot/period in a templated string appears to cause jinja template errors in 
> Airflow.
> example:
> I constructed a dictionary to pass as an environment into BashOperator
> i.e.
> envvars = {'use_this_script':'/Users/mshire/x.sh'}
> task4 = BashOperator(
>     bash_command="env",
>     task_id="test_env_variables",
>     env=envvars,
>     dag=dag
>     )
> If I remove the dot/period (i.e. ".") in "x.sh" then it works.  Otherwise I 
> get an error below.  I have tried it directly in python with jinja2 and it 
> works, but not in Airflow.
> > airflow test scratch test_env_variables 2016-10-01
> [2016-10-16 00:39:32,981] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-10-16 00:39:33,374] {models.py:154} INFO - Filling up the DagBag from 
> /Users/mshire/airflow/dags
> envvars= {'use_this_script': '/Users/mshire/x.sh'}
> [2016-10-16 00:39:33,479] {models.py:1196} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2016-10-16 00:39:33,480] {models.py:1219} INFO - Executing 
> <Task(BashOperator): test_env_variables> on 2016-10-01 00:00:00
> [2016-10-16 00:39:33,489] {models.py:1286} ERROR - /Users/mshire/x.sh
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1233, 
> in run
>     self.render_templates()
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1409, 
> in render_templates
>     rendered_content = rt(attr, content, jinja_context)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2019, 
> in render_template
>     return self.render_template_from_field(attr, content, context, jinja_env)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
> in render_template_from_field
>     for k, v in list(content.items())}
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, 
> in <dictcomp>
>     for k, v in list(content.items())}
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2017, 
> in render_template
>     return jinja_env.get_template(content).render(**context)
>   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
> 812, in get_template
>     return self._load_template(name, self.make_globals(globals))
>   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 
> 774, in _load_template
>     cache_key = self.loader.get_source(self, name)[1]
>   File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, 
> in get_source
>     raise TemplateNotFound(template)
> TemplateNotFound: /Users/mshire/x.sh
> [2016-10-16 00:39:33,490] {models.py:1306} INFO - Marking task as FAILED.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to