k8s as master would be nice but doesn’t solve the problem of running the full cluster and is an orthogonal issue.
We’d like to deploy Spark Workers/Executors and Master (whatever master is easiest to talk about since we really don’t care) in pods as we do with the other services we use. Replace Spark Master with k8s if you insist. How do the executors get deployed? We have our own containers that almost work for 2.3.3. We have used this before with older Spark so we are reasonably sure it makes sense. We just wonder if our own image builds and charts are the best starting point. Does anyone have something they like? From: Matt Cheah <mch...@palantir.com> <mch...@palantir.com> Reply: Matt Cheah <mch...@palantir.com> <mch...@palantir.com> Date: July 1, 2019 at 4:45:55 PM To: Pat Ferrel <p...@occamsmachete.com> <p...@occamsmachete.com>, user@spark.apache.org <user@spark.apache.org> <user@spark.apache.org> Subject: Re: k8s orchestrating Spark service Sorry, I don’t quite follow – why use the Spark standalone cluster as an in-between layer when one can just deploy the Spark application directly inside the Helm chart? I’m curious as to what the use case is, since I’m wondering if there’s something we can improve with respect to the native integration with Kubernetes here. Deploying on Spark standalone mode in Kubernetes is, to my understanding, meant to be superseded by the native integration introduced in Spark 2.4. *From: *Pat Ferrel <p...@occamsmachete.com> *Date: *Monday, July 1, 2019 at 4:40 PM *To: *"user@spark.apache.org" <user@spark.apache.org>, Matt Cheah < mch...@palantir.com> *Subject: *Re: k8s orchestrating Spark service Thanks Matt, Actually I can’t use spark-submit. We submit the Driver programmatically through the API. But this is not the issue and using k8s as the master is also not the issue though you may be right about it being easier, it doesn’t quite get to the heart. We want to orchestrate a bunch of services including Spark. The rest work, we are asking if anyone has seen a good starting point for adding Spark as a k8s managed service. From: Matt Cheah <mch...@palantir.com> <mch...@palantir.com> Reply: Matt Cheah <mch...@palantir.com> <mch...@palantir.com> Date: July 1, 2019 at 3:26:20 PM To: Pat Ferrel <p...@occamsmachete.com> <p...@occamsmachete.com>, user@spark.apache.org <user@spark.apache.org> <user@spark.apache.org> Subject: Re: k8s orchestrating Spark service I would recommend looking into Spark’s native support for running on Kubernetes. One can just start the application against Kubernetes directly using spark-submit in cluster mode or starting the Spark context with the right parameters in client mode. See https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org] <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=4XyH4cxucBNQAlSaHyR4gXJbHIo9g9vcur4VzBCYkwk&s=Q6mv_pZUq3UmxJU6EiJYJvG8L44WBeWJyAnw3vG0GBw&e=> I would think that building Helm around this architecture of running Spark applications would be easier than running a Spark standalone cluster. But admittedly I’m not very familiar with the Helm technology – we just use spark-submit. -Matt Cheah *From: *Pat Ferrel <p...@occamsmachete.com> *Date: *Sunday, June 30, 2019 at 12:55 PM *To: *"user@spark.apache.org" <user@spark.apache.org> *Subject: *k8s orchestrating Spark service We're trying to setup a system that includes Spark. The rest of the services have good Docker containers and Helm charts to start from. Spark on the other hand is proving difficult. We forked a container and have tried to create our own chart but are having several problems with this. So back to the community… Can anyone recommend a Docker Container + Helm Chart for use with Kubernetes to orchestrate: - Spark standalone Master - several Spark Workers/Executors This not a request to use k8s to orchestrate Spark Jobs, but the service cluster itself. Thanks
