[
https://issues.apache.org/jira/browse/AIRFLOW-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lightning Analytics updated AIRFLOW-6384:
-----------------------------------------
Summary: Custom Spark Livy Operator (was: Cusomt Spark Livy Operator)
> Custom Spark Livy Operator
> --------------------------
>
> Key: AIRFLOW-6384
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6384
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.5, 1.10.6
> Reporter: Lightning Analytics
> Priority: Minor
>
> Currently, Livy Rest API to submit a job to spark cluster is not available.
> We have developed a Spark Livy Operator and thoroughly tested by submitting
> spark jobs to the cluster using the Livy Rest API. Livy provides two sessions
> namely, Batch and Interactive. The custom Spark Livy Operator developed by us
> provides Interactive session submission and additionally sends heart beats to
> check the status of the batch. Extensive testing has been performed using
> Celery executor, and all available hooks and libraries in Airflow have been
> integrated.
> We would like to publish the operator to the Apache open source community for
> enhancement, use and distribution. Please inform about the steps for creating
> a Git pull request for Airflow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
