[
https://issues.apache.org/jira/browse/AIRFLOW-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044018#comment-17044018
]
Kaxil Naik commented on AIRFLOW-6384:
-------------------------------------
This was added in
https://github.com/apache/airflow/commit/d3cf23dc07b5fb92ee2a5be07b0685a4fca36f86
*Jira*: https://issues.apache.org/jira/browse/AIRFLOW-5470
*PR*: https://github.com/apache/airflow/pull/6090
> Custom Spark Livy Operator
> --------------------------
>
> Key: AIRFLOW-6384
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6384
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.5, 1.10.6
> Reporter: Lightning Analytics
> Priority: Minor
>
> Currently, Livy Rest API to submit a job to spark cluster is not available.
> We have developed a Spark Livy Operator and thoroughly tested by submitting
> spark jobs to the cluster using the Livy Rest API. Livy provides two sessions
> namely, Batch and Interactive. The custom Spark Livy Operator developed by us
> provides Interactive session submission and additionally sends heart beats to
> check the status of the batch. Extensive testing has been performed using
> Celery executor, and all available hooks and libraries in Airflow have been
> integrated.
> We would like to publish the operator to the Apache open source community for
> enhancement, use and distribution. Please inform about the steps for creating
> a Git pull request for Airflow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
