> It's applicable regardless of if the operators are maintained as part of
> Spark core or not, with the maturity of Kubernetes features around CRD
> support and webhooks. The GCP Spark operator supports a lot of additional
> pod/container configs using a webhook, and this approach seems pretty
> s
Folks have (correctly) pointed out that an operator does not need to be
coupled to the Apache Spark project. However, I believe there are some
strategic community benefits to supporting a Spark operator that should be
weighed against the costs of maintaining one.
*) The Kubernetes ecosystem is evo
Hi Erik,
I agree with what you said about the community benefits of supporting
operators for Spark. However, that doesn't necessarily mean the operators
need and should be part of Spark core.
> *) The Kubernetes ecosystem is evolving toward adopting operators as the
de facto standard for deployin
+1. This and the GCP Spark Operator, although being very useful for k8s
users, are not something needed by all Spark users, not even by all Spark
on k8s users.
On Thu, Oct 10, 2019 at 6:34 PM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> wrote:
> Hi all,
>
> I also left a comment on t
Hi all,
I also left a comment on the PR with more details. I dont see why the java
operator should be maintained by the Spark project.
This is an interesting project and could thrive on its own as an external
operator project.
Best,
Stavros
On Thu, Oct 10, 2019 at 7:51 PM Sean Owen wrote:
> I'
I'd have the same question on the PR - why does this need to be in the
Apache Spark project vs where it is now? Yes, it's not a Spark package
per se, but it seems like this is a tool for K8S to use Spark rather
than a core Spark tool.
Yes of course all the packages, licenses, etc have to be overha
Hello,
Spark Operator is a tool that can deploy/scale and help with monitoring of
Spark clusters on Kubernetes. It follows the operator pattern [1]
introduced by CoreOS so it watches for changes in custom resources
representing the desired state of the clusters and does the steps to
achieve this s
