[jira] (AIRFLOW-802) Integration of spark-submit

[Fokko Driesprong (JIRA)]

Title: Message Title

Fokko Driesprong updated an issue   Apache Airflow / AIRFLOW-802 Integration of spark-submit Change By: Fokko Driesprong Summary: Integration of  PySpark hook  spark-submit Hi  guys  all , I would like to  contribute  add  the  PySpark  spark-submit operator and  hook  to natively support Apache Spark pyspark jobs . Right now  we  only  support spark-sql operations by  the  SparkSqlOperator. Since  spark- submit is a different binary, I've created a new hook and operator. Besides the fact that spark-submit has much more options to configure, there might be features in the future that share functionality with the spark- sql operator . One main implementation difference between the spark-sql and the spark-submit operator  is  supported  the use of the connection_id of Airflow .  It accepts a connection to set the master and the queue. Cheers , Fokko ! Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)