[
https://issues.apache.org/jira/browse/AIRFLOW-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesh Sivalingam updated AIRFLOW-1918:
---------------------------------------
Description:
When creating a docker operator, and providing the command parameter like so:
{code: python}
create_cluster = DockerOperator(
image="my/image",
command="bash /home/ec2-user/create-cluster.sh",
cpus=1
{code}
I get the following error:
{code:java}
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting
attempt 1 of 1
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
[2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,639] {models.py:1342} INFO - Executing <Task(DockerOperator):
create_flintrock_cluster> on 2017-12-13 13:21:20.147853
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,652] {models.py:1417} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:
TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,653] {models.py:1441} INFO - Marking task as FAILED.
[2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,667] {models.py:1462} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/bin/airflow", line 28, in <module>
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
args.func(args)
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
pool=args.pool,
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in
wrapper
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: result =
func(*args, **kwargs)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
jinja2.exceptions.TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
{code}
Within the DockerOperator code, if I comment out line 79 ( {{template_fields =
('command',)}}) then the task completes successfully.
was:
When creating a docker operator, and providing the command parameter like so:
{code: python}
create_cluster = DockerOperator(
image="my/image",
command="bash /home/ec2-user/create-cluster.sh",
cpus=1
{code}
I get the following error:
{code:java}
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting
attempt 1 of 1
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
[2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,639] {models.py:1342} INFO - Executing <Task(DockerOperator):
create_flintrock_cluster> on 2017-12-13 13:21:20.147853
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,652] {models.py:1417} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:
TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,653] {models.py:1441} INFO - Marking task as FAILED.
[2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask: [2017-12-13
13:21:22,667] {models.py:1462} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/bin/airflow", line 28, in <module>
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
args.func(args)
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
pool=args.pool,
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in
wrapper
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: result =
func(*args, **kwargs)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
self.render_templates()
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
render_templates
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
render_template
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: return
jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
get_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: return
self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
_load_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
get_source
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: raise
TemplateNotFound(template)
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
jinja2.exceptions.TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
{code}
Within the DockerOperator code, if I comment out line 79 (`template_fields =
('command',)`) then the task completes successfully.
> DockerOperator forcing Jinja templating on "command" when not required
> ----------------------------------------------------------------------
>
> Key: AIRFLOW-1918
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1918
> Project: Apache Airflow
> Issue Type: Bug
> Components: docker
> Affects Versions: Airflow 1.8
> Reporter: Ganesh Sivalingam
>
> When creating a docker operator, and providing the command parameter like so:
> {code: python}
> create_cluster = DockerOperator(
> image="my/image",
> command="bash /home/ec2-user/create-cluster.sh",
> cpus=1
> {code}
> I get the following error:
> {code:java}
> [2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting
> attempt 1 of 1
> [2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
> [2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask:
> [2017-12-13 13:21:22,639] {models.py:1342} INFO - Executing
> <Task(DockerOperator): create_flintrock_cluster> on 2017-12-13 13:21:20.147853
> [2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask:
> [2017-12-13 13:21:22,652] {models.py:1417} ERROR - bash
> /home/ec2-user/create-cluster.sh
> [2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
> self.render_templates()
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
> render_templates
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
> rendered_content = rt(attr, content, jinja_context)
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
> render_template
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
> jinja_env.get_template(content).render(**context)
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
> get_template
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return
> self._load_template(name, self.make_globals(globals))
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
> _load_template
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:
> cache_key = self.loader.get_source(self, name)[1]
> [2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
> get_source
> [2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: raise
> TemplateNotFound(template)
> [2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:
> TemplateNotFound: bash /home/ec2-user/create-cluster.sh
> [2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:
> [2017-12-13 13:21:22,653] {models.py:1441} INFO - Marking task as FAILED.
> [2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask:
> [2017-12-13 13:21:22,667] {models.py:1462} ERROR - bash
> /home/ec2-user/create-cluster.sh
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/bin/airflow", line 28, in <module>
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
> args.func(args)
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:
> pool=args.pool,
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in
> wrapper
> [2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: result
> = func(*args, **kwargs)
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
> self.render_templates()
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in
> render_templates
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:
> rendered_content = rt(attr, content, jinja_context)
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in
> render_template
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: return
> jinja_env.get_template(content).render(**context)
> [2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in
> get_template
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: return
> self._load_template(name, self.make_globals(globals))
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in
> _load_template
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
> cache_key = self.loader.get_source(self, name)[1]
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in
> get_source
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: raise
> TemplateNotFound(template)
> [2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:
> jinja2.exceptions.TemplateNotFound: bash /home/ec2-user/create-cluster.sh
> [2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
> {code}
> Within the DockerOperator code, if I comment out line 79 ( {{template_fields
> = ('command',)}}) then the task completes successfully.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
