Rob Keevil created AIRFLOW-2140:

             Summary: Add Kubernetes to Spark Submit Operator
                 Key: AIRFLOW-2140
             Project: Apache Airflow
          Issue Type: New Feature
    Affects Versions: 1.9.0
            Reporter: Rob Keevil
            Assignee: Rob Keevil

Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the existing 
Standalone, Yarn and Mesos resource managers.

We should extend the spark submit operator to enable the new K8s spark submit 
options, and to be able to monitor Spark jobs running within Kubernetes.

I already have working code for this, I need to test the monitoring/log parsing 
code and make sure that Airflow is able to terminate Kubernetes pods when jobs 
are cancelled etc.

This message was sent by Atlassian JIRA

Reply via email to