Rob Keevil created AIRFLOW-2140:
-----------------------------------

             Summary: Add Kubernetes to Spark Submit Operator
                 Key: AIRFLOW-2140
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2140
             Project: Apache Airflow
          Issue Type: New Feature
    Affects Versions: 1.9.0
            Reporter: Rob Keevil
            Assignee: Rob Keevil


Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the existing 
Standalone, Yarn and Mesos resource managers. 

https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md

We should extend the spark submit operator to enable the new K8s spark submit 
options, and to be able to monitor Spark jobs running within Kubernetes.

I already have working code for this, I need to test the monitoring/log parsing 
code and make sure that Airflow is able to terminate Kubernetes pods when jobs 
are cancelled etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to