Ace Haidrey created AIRFLOW-1770:
------------------------------------
Summary: Add option to file and hiveconfs in HiveOperator
Key: AIRFLOW-1770
URL: https://issues.apache.org/jira/browse/AIRFLOW-1770
Project: Apache Airflow
Issue Type: Improvement
Reporter: Ace Haidrey
Assignee: Ace Haidrey
The HiveOperator as it currently stands is not flexible enough to accept a hive
file and operate on that. You need to read in the contents and pass it and if
you do that you need to change the way hiveconfs are in your file to jinja
templating.
Many teams already have their existing sql/hql files and don't want to convert
them to make them as portable and decoupled as possible.
To accomplish this all we need to do is add the option to pass a hql_file and
hiveconfs to the HiveOperator. We change the code in the execute to throw an
error if both a hql_file and an hql statement are passed. If just hql_file the
simplest way without changing the code of the hive hook is to just read the
content of the hql_file and set it to be the self.hql. The hiveconfs get passed
directly to the run_cli method and we can combine them with the already passed
in hiveconfs.
If we want to make it optional to pass in the context as hiveconfs we can add
that too as related to AIRFLOW-788.
I've included some simple tests to show it all works how we expect.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
