[
https://issues.apache.org/jira/browse/AIRFLOW-240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Reda updated AIRFLOW-240:
------------------------------
Description:
When using the PostgresOperator, I've been unable to get templating to work.
It's quite possible I'm doing something wrong, but I'm pretty lost as to what
the issue might be. Here's an example to reproduce the TemplateNotFound error
I've been getting:
{code:title=/my_dag/csv_to_redshift.py}
copy_s3_to_redshift = PostgresOperator(
task_id='load_table',
sql=this_dag_path + '/copy_to_redshift.sql',
params=dict(
AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
),
postgres_conn_id='postgres_redshift',
autocommit=False,
dag=dag
)
{code}
{code:title=/my_dag/copy_to_redshift.sql}
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{
AWS_SECRET_ACCESS_KEY }}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}
{code}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
in resolve_template_files
setattr(self, attr, env.loader.get_source(env, content)[0])
File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in
get_source
raise TemplateNotFound(template)
TemplateNotFound:
/home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
{code}
Any ideas towards a fix are much appreciated.
was:
When using the PostgresOperator, I've been unable to get templating for
instance. Here's an example to reproduce the error:
{code:title=/my_dag/csv_to_redshift.py}
copy_s3_to_redshift = PostgresOperator(
task_id='load_table',
sql=this_dag_path + '/copy_to_redshift.sql',
params=dict(
AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
),
postgres_conn_id='postgres_redshift',
autocommit=False,
dag=dag
)
{code}
{code:title=/my_dag/copy_to_redshift.sql}
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{
AWS_SECRET_ACCESS_KEY }}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}
{code}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
in resolve_template_files
setattr(self, attr, env.loader.get_source(env, content)[0])
File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in
get_source
raise TemplateNotFound(template)
TemplateNotFound:
/home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
{code}
Any ideas?
> TemplateNotFound Error when using PostgresOperator and SQL script
> -----------------------------------------------------------------
>
> Key: AIRFLOW-240
> URL: https://issues.apache.org/jira/browse/AIRFLOW-240
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Environment: Ubuntu 14.04
> Reporter: Greg Reda
>
> When using the PostgresOperator, I've been unable to get templating to work.
> It's quite possible I'm doing something wrong, but I'm pretty lost as to what
> the issue might be. Here's an example to reproduce the TemplateNotFound error
> I've been getting:
> {code:title=/my_dag/csv_to_redshift.py}
> copy_s3_to_redshift = PostgresOperator(
> task_id='load_table',
> sql=this_dag_path + '/copy_to_redshift.sql',
> params=dict(
> AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
> AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
> ),
> postgres_conn_id='postgres_redshift',
> autocommit=False,
> dag=dag
> )
> {code}
> {code:title=/my_dag/copy_to_redshift.sql}
> COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
> CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID
> }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY }}'
> CSV
> NULL as 'null'
> IGNOREHEADER as 1;
> {code}
> {code}
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
> in resolve_template_files
> setattr(self, attr, env.loader.get_source(env, content)[0])
> File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187,
> in get_source
> raise TemplateNotFound(template)
> TemplateNotFound:
> /home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
> {code}
> Any ideas towards a fix are much appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
