Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
Ultimately I see this as being in the realm of Kubernetes knowledge, rather
than Spark knowledge. These features are well-documented in Kubernetes
documentation, and I am not sure there's a need
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@vanzin In general Kubernetes makes this super easy:
- The most basic workflow is to use the driver pod name (which is in the
output of Spark Submit, or can be found with `kubectl get pods
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@andrewor14 @liancheng can you chime in?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@foxish per SPARK-5176 and the associated
[PR](https://github.com/apache/spark/pull/4137), it seems there was a technical
issue with spark-internal that wouldn't allow Thrift Server to run on YARN
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@foxish @felixcheung Now verified my test runs successfully
https://github.com/apache-spark-on-k8s/spark-integration/pull/38
Github user ozzieba commented on a diff in the pull request:
https://github.com/apache/spark/pull/20272#discussion_r162108920
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -328,7 +328,7 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@foxish @felixcheung I wrote a
[test](https://github.com/ozzieba/spark-integration/commit/2c77c7d4ec9d1a82b0c097073eb71ebdfbac15b7),
but I'm having trouble with Minikube on Windows, and I can't run
Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
I'm getting stuck on
https://github.com/apache-spark-on-k8s/spark-integration/blob/master/integration-test/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala#L106
GitHub user ozzieba opened a pull request:
https://github.com/apache/spark/pull/20272
[SPARK-23078] [CORE] allow Spark Thrift Server to run in Kubernetes Cluster
mode
## What changes were proposed in this pull request?
allow Spark Thrift Server to run in Kubernetes Cluster
