Debashish Dhar created AIRFLOW-4188:
---------------------------------------
Summary: Customized LivySparkOperator
Key: AIRFLOW-4188
URL: https://issues.apache.org/jira/browse/AIRFLOW-4188
Project: Apache Airflow
Issue Type: New Feature
Components: DagRun
Reporter: Debashish Dhar
Airflow provides operators for various APIs, however there is no operator
available for Livy Rest API. Here at Deloitte we have developed and tested an
operator which can connect to Livy and submit jobs on EMR. Livy provides two
sessions, Batch and Interactive, the custom Livy spark operator provides
Interactive session submission as well as sends heart beats to check the batch
status. The operator has been successfully implemented at one of our current
client project.
We would like to publish our code to Airflow GitHub to enable further use,
research, change and distribution. The code is a standalone operator developed
using hooks and libraries present in the Airflow implementation. Please let us
know the next steps to create a Pull Request for the code and submit to the
open source community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
