[
https://issues.apache.org/jira/browse/AIRFLOW-240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Reda updated AIRFLOW-240:
------------------------------
Description:
When using the PostgresOperator, I've been unable to get templating for
instance. Here's an example to reproduce the error:
/my_dag/csv_to_redshift.py
----
copy_s3_to_redshift = PostgresOperator(
task_id='load_table',
sql=this_dag_path + '/copy_to_redshift.sql',
params=dict(
AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
),
postgres_conn_id='postgres_redshift',
autocommit=False,
dag=dag
)
/my_dag/copy_to_redshift.sql
{code:title=copy_to_redshift.sql}
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{
AWS_SECRET_ACCESS_KEY }}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
in resolve_template_files
setattr(self, attr, env.loader.get_source(env, content)[0])
File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in
get_source
raise TemplateNotFound(template)
TemplateNotFound:
/home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
Any ideas?
was:
When using the PostgresOperator, I've been unable to get templating for
instance. Here's an example to reproduce the error:
/my_dag/csv_to_redshift.py
----
copy_s3_to_redshift = PostgresOperator(
task_id='load_table',
sql=this_dag_path + '/copy_to_redshift.sql',
params=dict(
AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
),
postgres_conn_id='postgres_redshift',
autocommit=False,
dag=dag
)
/my_dag/copy_to_redshift.sql
----
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{
AWS_SECRET_ACCESS_KEY }}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
in resolve_template_files
setattr(self, attr, env.loader.get_source(env, content)[0])
File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in
get_source
raise TemplateNotFound(template)
TemplateNotFound:
/home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
Any ideas?
> TemplateNotFound Error when using PostgresOperator and SQL script
> -----------------------------------------------------------------
>
> Key: AIRFLOW-240
> URL: https://issues.apache.org/jira/browse/AIRFLOW-240
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Environment: Ubuntu 14.04
> Reporter: Greg Reda
>
> When using the PostgresOperator, I've been unable to get templating for
> instance. Here's an example to reproduce the error:
> /my_dag/csv_to_redshift.py
> ----
> copy_s3_to_redshift = PostgresOperator(
> task_id='load_table',
> sql=this_dag_path + '/copy_to_redshift.sql',
> params=dict(
> AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
> AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
> ),
> postgres_conn_id='postgres_redshift',
> autocommit=False,
> dag=dag
> )
> /my_dag/copy_to_redshift.sql
> {code:title=copy_to_redshift.sql}
> COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
> CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID
> }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY }}'
> CSV
> NULL as 'null'
> IGNOREHEADER as 1;
> {code}
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038,
> in resolve_template_files
> setattr(self, attr, env.loader.get_source(env, content)[0])
> File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187,
> in get_source
> raise TemplateNotFound(template)
> TemplateNotFound:
> /home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
> Any ideas?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
