Github user ozzieba commented on the issue:
https://github.com/apache/spark/pull/20272
@vanzin In general Kubernetes makes this super easy:
- The most basic workflow is to use the driver pod name (which is in the
output of Spark Submit, or can be found with `kubectl get pods`), and run
`kubectl port-forward spark-app-driver-podname 31416:10000`, which will
automatically
[forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
localhost:31416 to the pod's port 10000. Any jdbc application can then be used
to query `jdbc:hive2://localhost:31416`. This is the approach I took in the
integration tests linked above.
- Alternatively, any other application on the cluster can simply use
spark-app-driver-podname:10000, which will be resolved by
[kube-dns](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/).
- For persistent external access one can run `kubectl expose pod
spark-app-driver-podname --type=NodePort --port 10000` to create a Kubernetes
[Service](https://kubernetes.io/docs/concepts/services-networking/service/)
which will accept connections on a particular port of *every* node on the
cluster and send them to the driver's port 10000.
- On a cloud environment, using `type=LoadBalancer` in the above will
create a global [load
balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/)
that can be used to access Thrift from across the Internet.
- One can also define a service that automatically selects the driver using
a user-specified label (eg `bin/spark-submit.sh --conf
spark.kubernetes.driver.label.mythriftserver=true` followed by `kubectl expose
pod --selector=mythriftserver=true`)
Perhaps the key point is that these are all core Kubernetes features, and
not specific to Spark in any way. Users familiar with Kubernetes should be able
to find the approach that works best for their environment and use case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
