Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/18209
@manojlds I'm a part of the Spark-on-k8s team that's currently building k8s
integration for Spark outside of the Apache Spark repo. You can follow our
work at https://github.com/apache-spark-on-k8s/spark
Here's the current outstanding diff on top of `apache/branch-2.1`:
https://github.com/apache-spark-on-k8s/spark/pull/200 , produced from about
6months of work in a varied group from approaching a dozen companies at this
point. We aim to eventually bring this into the Apache repo in some form,
though the timeline for that is not yet fixed.
You can follow from the original design and discussion for the kubernetes
efforts at https://issues.apache.org/jira/browse/SPARK-18278
One of the major pieces of feedback we received back in November was that,
if possible, the kubernetes integration should be done as a plugin to the
Apache Spark repo. Unfortunately that plugin point does not yet exist in
Spark. So we filed https://issues.apache.org/jira/browse/SPARK-19700 to begin
the process of creating that API but have not yet devoted effort to building
that plugin point. Once active development on the k8s integration begin to
slow down and near completion (starting to see signs of this slowing this
month) we'll probably shift focus to the plugin point as a means of gaining
even wider use of the k8s integration. On what a plugin point could possibly
look like, we produced a thought experiment at
https://github.com/palantir/spark/pull/81 which you may find interesting.
As of now, it remains unclear whether the k8s code will be brought in as a
first-class citizen alongside Mesos and YARN, or whether Spark will gain a
plugin point and the k8s integration.
We'd be happy to be a part of any wider efforts to create a plugin point
for custom resource managers at SPARK-19700. Is that something the Nomad team
would be interested in contributing to?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
