[
https://issues.apache.org/jira/browse/AIRFLOW-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090379#comment-17090379
]
Rajesh commented on AIRFLOW-4188:
---------------------------------
Hi , we have similar requirement where we have to submit EMR jobs via Airflow
through Livy operator.
Please let me know If any reliable official airflow Livy operator provided.
> Customized LivySparkOperator
> ----------------------------
>
> Key: AIRFLOW-4188
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4188
> Project: Apache Airflow
> Issue Type: New Feature
> Components: DagRun
> Reporter: Debashish Dhar
> Priority: Minor
> Labels: features
>
> Airflow provides operators for various APIs, however there is no operator
> available for Livy Rest API. Here at Deloitte we have developed and tested an
> operator which can connect to Livy and submit jobs on EMR. Livy provides two
> sessions, Batch and Interactive, the custom Livy spark operator provides
> Interactive session submission as well as sends heart beats to check the
> batch status. The operator has been successfully implemented at one of our
> current client project.
> We would like to publish our code to Airflow GitHub to enable further use,
> research, change and distribution. The code is a standalone operator
> developed using hooks and libraries present in the Airflow implementation.
> Please let us know the next steps to create a Pull Request for the code and
> submit to the open source community.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
