[
https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472035#comment-16472035
]
Abhishek Rao commented on SPARK-24179:
--
We have brought up Spark History Server on Kubernetes using following approach.
1) Prepare History-server docker container.
a. Inherit Dockerfile from Kubespark/spark-base.
b. Set Environment variable SPARK_NO_DAEMONIZE so that history-server can
start as a Daemon.
c. As part of Docker entry point, we’re invoking start-history-server.sh
2) Bringup Kubernetes pod using the docker image built in step 1
3) Bringup Kubernetes service for the histoty-server pod with type ClusterIP or
NodePort.
4) If we bringup service with NodePort, History server can be accessed using
NodeIP:NodePort
5) If we bringup service with ClusterIP, then we need to create Kubernetes
ingress which will forward the request to the service brought up in step 3.
Once we have the ingress up, history server can be accessed using Edge Node IP
with the “path” specified in ingress.
6) Only limitation with ingress is that redirection is happening only when we
use “/” in path. If we use any other string, redirection is not happening
This is validated using spark binaries from apache-spark-on-k8s forked from
spark 2.2 as well as apache spark 2.3. Attached are the screen shots for both
versions of spark.
> History Server for Kubernetes
> -
>
> Key: SPARK-24179
> URL: https://issues.apache.org/jira/browse/SPARK-24179
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Eric Charles
>Priority: Major
> Attachments: Spark2_2_History_Server.PNG, Spark2_3_History_Server.PNG
>
>
> The History server is missing when running on Kubernetes, with the side
> effect we can not debug post-mortem or analyze after-the-fact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
