[
https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960443#comment-16960443
]
Aliaksandr Sasnouskikh commented on LIVY-588:
---------------------------------------------
This epic fixes that ticket splitting it to multiple steps:
https://issues.apache.org/jira/browse/LIVY-702
> Full support for Spark on Kubernetes
> ------------------------------------
>
> Key: LIVY-588
> URL: https://issues.apache.org/jira/browse/LIVY-588
> Project: Livy
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Environment: Kubernetes
> Reporter: Aliaksandr Sasnouskikh
> Priority: Major
>
>
> Since Spark on Kubernetes has been released relatively long ago this can be a
> good idea to include Kubernetes support to Livy project as well, as it can
> solve much problems related to working with Spark on Kubernetes, it can fully
> replace Yarn in case of working atop Kubernetes cluster:
> - Livy UI has cached logs/diagnostics page
> - Livy UI shows links to Spark UI and Spark History Server
> - With Kubernetes Ingress resource Livy can be configured to serve as an
> orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
> option to create routes to Spark UI)
> - Nginx Ingress solves `basePath` support for Spark UI and History Server as
> well as has lots of auth integrations available:
> [https://github.com/kubernetes/ingress-nginx]
> - Livy UI can be integrated with Grafana Loki logs (PR provides solution for
> that)
> Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
> Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
> Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
> Design concept:
> [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md]
> Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes
> v1.11.8:
> - Image: Spark 2.4.3 with Hadoop 3.2.0
> ([https://github.com/jahstreet/spark-on-kubernetes-docker])
> - History Server:
> [https://github.com/helm/charts/tree/master/stable/spark-history-server]
> - Jupyter Notebook with Sparkmagic:
> [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter]
> Unit and/or integration tests are in progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
