Ruslan Dautkhanov created AIRFLOW-1012:
------------------------------------------
Summary: Add run_as_script option so jinja templating can be used
for sql parameter
Key: AIRFLOW-1012
URL: https://issues.apache.org/jira/browse/AIRFLOW-1012
Project: Apache Airflow
Issue Type: Improvement
Components: core, db
Affects Versions: Airflow 1.8
Reporter: Ruslan Dautkhanov
Assignee: Ruslan Dautkhanov
It would be great to extend jinja templating to sql parameter for SQL Operators.
With this improvement, it's possible to have extended Jinja template like below
that generates multiple SQL statements that can be passed as a single 'sql'
parameter, separated by ';' separator:
{noformat}
)
>> OracleOperator( task_id='give_owner_grants', oracle_conn_id=ora_conn1,
run_as_script=True,
sql='''
{% for role in ['CONNECT', 'RESOURCE'] %}
GRANT {{ role }} TO {{ schema }};
{% endfor %}
{% for create_grant in ['PROCEDURE', 'SEQUENCE', 'SESSION', 'TABLE',
'VIEW'] %}
GRANT CREATE {{ create_grant }} TO {{ schema }};
{% endfor %}
{% for tbsp in ['DISCOVER_MART_IDX01', 'DISCOVER_MART_TBS01',
'STAGING_NOLOG'] %}
ALTER USER {{ schema }} QUOTA UNLIMITED ON {{ tbsp }};
{% endfor %}
GRANT SELECT ANY TABLE TO {{ schema }};
GRANT EXECUTE ON SYS.DBMS_SESSION TO {{ schema }};
'''
)
>> DummyOperator(task_id='stop')
{noformat}
Notice there are three Jinja 'for' loops that generate multiple SQL DDL
statements.
Without this change, sql has to be passed as an Python array, and Jinja
templating can't be used.
I've tested this change with OracleOperator and works as expected.
Notice `run_as_script=True` parameter. run_as_script defaults to False so this
is a backward-compatible change.
Most of the change is in airflow/hooks/dbapi_hook.py (very straightforward as
run() already supports running an array of statements) and a light change of
airflow/operators/oracle_operator.py - so this change can be easily applied to
other sql operators.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
