Rob Keevil created AIRFLOW-2140:
-----------------------------------
Summary: Add Kubernetes to Spark Submit Operator
Key: AIRFLOW-2140
URL: https://issues.apache.org/jira/browse/AIRFLOW-2140
Project: Apache Airflow
Issue Type: New Feature
Affects Versions: 1.9.0
Reporter: Rob Keevil
Assignee: Rob Keevil
Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the existing
Standalone, Yarn and Mesos resource managers.
https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md
We should extend the spark submit operator to enable the new K8s spark submit
options, and to be able to monitor Spark jobs running within Kubernetes.
I already have working code for this, I need to test the monitoring/log parsing
code and make sure that Airflow is able to terminate Kubernetes pods when jobs
are cancelled etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
