Roi Teveth created AIRFLOW-6542:
-----------------------------------
Summary: sparkKubernetes operator for
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
Key: AIRFLOW-6542
URL: https://issues.apache.org/jira/browse/AIRFLOW-6542
Project: Apache Airflow
Issue Type: Improvement
Components: contrib
Affects Versions: 1.10.7
Reporter: Roi Teveth
hi,
we working on spark on Kubernetes POC using the google cloud platform
spark-k8s-operator
[https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] and haven't
found native airflow integration for it so we wrote one:
kubernetes_hook which create and get kuberenetes crd object
spark_kubernetes_operator which sends sparkapplication crd to kubernetes cluster
spark_kubernetes_sensor which poke sparkapplication state
operator example with spark-pi
application:[https://github.com/roitvt/airflow-spark-on-k8s-operator]
I'll be glad to contribute our operator to airflow contrib
Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
