[
https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929591#comment-15929591
]
Andrew Ash commented on SPARK-18278:
------------------------------------
As an update on this ticket:
For those not already aware, work on native Spark integration with Kubernetes
has been proceeding for the past several months in this repo
https://github.com/apache-spark-on-k8s/spark in the {{branch-2.1-kubernetes}}
branch, based off the 2.1.0 Apache release.
We have an active core of about a half dozen contributors to the project with a
wider group observing of about another dozen. Communication happens through
the issues on the GitHub repo, a dedicated room in the Kubernetes Slack, and
weekly video conferences hosted by the Kubernetes Big Data SIG.
The full patch set is currently about 5500 lines, with about 500 of that as
user/dev documentation. Infrastructure-wise, we have a cloud-hosted CI Jenkins
instance set up donated by project members, which is running both unit tests
and Kubernetes integration tests over the code.
We recently entered a code freeze for our release branch and are preparing a
first release to the wider community, which we plan to announce on the general
Spark users list. It includes the completed "phase one" portion of the design
doc shared a few months ago
(https://docs.google.com/document/d/1_bBzOZ8rKiOSjQg78DXOA3ZBIo_KkDJjqxVuq0yXdew/edit#heading=h.fua3ml5mcolt),
featuring cluster mode with static allocation of executors, submission of
local resources, SSL throughout, and support for JVM languages (Java/Scala).
After that release we'll be continuing to stabilize and improve the phase one
feature set and move into a second phase of kubernetes work. It will likely be
focused on support for dynamic allocation, though we haven't finalized planning
for phase two yet. Working on the pluggable scheduler in SPARK-19700 may be
included as well.
Interested parties are of course welcome to watch the repo, join the weekly
video conferences, give the code a shot, and contribute to the project!
> Support native submission of spark jobs to a kubernetes cluster
> ---------------------------------------------------------------
>
> Key: SPARK-18278
> URL: https://issues.apache.org/jira/browse/SPARK-18278
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Deploy, Documentation, Scheduler, Spark Core
> Reporter: Erik Erlandson
> Attachments: SPARK-18278 - Spark on Kubernetes Design Proposal.pdf
>
>
> A new Apache Spark sub-project that enables native support for submitting
> Spark applications to a kubernetes cluster. The submitted application runs
> in a driver executing on a kubernetes pod, and executors lifecycles are also
> managed as pods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
