[
https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr Sasnouskikh updated LIVY-588:
----------------------------------------
Description:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
* Livy UI has cached logs/diagnostics page
* Livy UI shows links to Spark UI and Spark History Server
* With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
* Nginx Ingress solves {{basePath}} support for Spark UI and History Server as
well as has lots of auth integrations available:
[https://github.com/kubernetes/ingress-nginx]
Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
Associated PR: [https://github.com/apache/incubator-livy/pull/167]
Design concept doc:
[https://github.com/jahstreet/spark-on-kubernetes-helm/blob/master/README.md]
was:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
* Livy UI has cached logs/diagnostics page
* Livy UI shows links to Spark UI and Spark History Server
* With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
* Nginx Ingress solves {{basePath}} support for Spark UI and History Server as
well as has lots of auth integrations available:
[https://github.com/kubernetes/ingress-nginx]
Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
Associated PR: [https://github.com/apache/incubator-livy/pull/167]
> Full support for Spark on Kubernetes
> ------------------------------------
>
> Key: LIVY-588
> URL: https://issues.apache.org/jira/browse/LIVY-588
> Project: Livy
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Environment: Kubernetes
> Reporter: Aliaksandr Sasnouskikh
> Priority: Major
>
> Since Spark on Kubernetes has been released relatively long ago this can be a
> good idea to include Kubernetes support to Livy project as well, as it can
> solve much problems related to working with Spark on Kubernetes, it can fully
> replace Yarn in case of working atop Kubernetes cluster:
> * Livy UI has cached logs/diagnostics page
> * Livy UI shows links to Spark UI and Spark History Server
> * With Kubernetes Ingress resource Livy can be configured to serve as an
> orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
> option to create routes to Spark UI)
> * Nginx Ingress solves {{basePath}} support for Spark UI and History Server
> as well as has lots of auth integrations available:
> [https://github.com/kubernetes/ingress-nginx]
> Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
> Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
> Associated PR: [https://github.com/apache/incubator-livy/pull/167]
> Design concept doc:
> [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/master/README.md]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
