[
https://issues.apache.org/jira/browse/AIRFLOW-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572855#comment-16572855
]
ASF subversion and git services commented on AIRFLOW-2140:
----------------------------------------------------------
Commit f58246d2ef265eb762c179a12c40e011ce62cea1 in incubator-airflow's branch
refs/heads/v1-10-test from [~ashb]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f58246d ]
[AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook (#3700)
This extra dep is a quasi-breaking change when upgrading - previously
there were no deps outside of Airflow itself for this hook. Importing
the k8s libs breaks installs that aren't also using Kubernetes.
This makes the dep optional for anyone who doesn't explicitly use the
functionality
(cherry picked from commit 0be002eebb182b607109a0390d7f6fb8795c668b)
Signed-off-by: Bolke de Bruin <bo...@xs4all.nl>
> Add Kubernetes Scheduler to Spark Submit Operator
> -------------------------------------------------
>
> Key: AIRFLOW-2140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2140
> Project: Apache Airflow
> Issue Type: New Feature
> Affects Versions: 1.9.0
> Reporter: Rob Keevil
> Assignee: Rob Keevil
> Priority: Major
> Fix For: 2.0.0
>
>
> Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the
> existing Standalone, Yarn and Mesos resource managers.
> https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md
> We should extend the spark submit operator to enable the new K8s spark submit
> options, and to be able to monitor Spark jobs running within Kubernetes.
> I already have working code for this, I need to test the monitoring/log
> parsing code and make sure that Airflow is able to terminate Kubernetes pods
> when jobs are cancelled etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
