Boris Tyukin created AIRFLOW-1713:
-------------------------------------
Summary: Return results optionally from spark_sql_hook
Key: AIRFLOW-1713
URL: https://issues.apache.org/jira/browse/AIRFLOW-1713
Project: Apache Airflow
Issue Type: Improvement
Components: contrib
Affects Versions: 1.8.1
Reporter: Boris Tyukin
Assignee: Boris Tyukin
Priority: Minor
spark_sql_hook is very useful to execute Spark SQL queries without much of
boilerplate with spark-submit but right now it is not possible to capture
execution results. One example if one wants to run "select count(1) from table"
and get that count back to airflow dag. To address that, optional argument with
regex expression is added to spark_sql_hook that will capture all spark log
output matching regex and return back to airflow.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
