Ganesh Sivalingam created AIRFLOW-1918:
------------------------------------------
Summary: DockerOperator forcing Jinja templating on "command" when
not required
Key: AIRFLOW-1918
URL: https://issues.apache.org/jira/browse/AIRFLOW-1918
Project: Apache Airflow
Issue Type: Bug
Components: docker
Affects Versions: Airflow 1.8
Reporter: Ganesh Sivalingam
When creating a docker operator, and providing the command parameter like so:
{code: python}
create_cluster = DockerOperator(
image="my/image",
command="bash /home/ec2-user/create-cluster.sh",
cpus=1
{code}
I get the following error:
{code:java}
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting
attempt 1 of 1
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
[2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,639] {models.py:1342} INFO - Executing <Task(DockerOperator):
create_flintrock_cluster> on 2017-12-13 13:21:20.147853
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,652] {models.py:1417} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:
TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,653] {models.py:1441} INFO - Marking task as FAILED.
[2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,667] {models.py:1462} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/bin/airflow", line 28, in <module>
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
args.func(args)
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
pool=args.pool,
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in
wrapper
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: result =
func(*args, **kwargs)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
jinja2.exceptions.TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
{code}
Within the DockerOperator code, if I comment out line 79 (`template_fields =
('command',)`) then the task completes successfully.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
