Felipe Lolas created AIRFLOW-3863:
-------------------------------------
Summary: Make SparkSubmitHook capable of executing spark-submit
through SSH Connection
Key: AIRFLOW-3863
URL: https://issues.apache.org/jira/browse/AIRFLOW-3863
Project: Apache Airflow
Issue Type: Improvement
Components: hooks, operators
Reporter: Felipe Lolas
Assignee: Felipe Lolas
Hi!
I want to add a functionality in SparkSubmitHook; connect to a remote server
though SSH and execute spark-submit. This would be great when working with
multiple spark clusters with an edge node for each cluster and installing a
airflow's worker onto edge node is not possibly.
Im currently implementing, but I wanna hear some thoughts from the airflow
community about the solution and if should be commited or not!
Cheers!
Felipe
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
