[jira] [Updated] (LIVY-703) Show SparkUI link on Livy UI when running on Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-703: Summary: Show SparkUI link on Livy UI when running on Kubernetes (was: Show SparkUI link on Livy UI while session is running) > Show SparkUI link on Livy UI when running on Kubernetes > --- > > Key: LIVY-703 > URL: https://issues.apache.org/jira/browse/LIVY-703 > Project: Livy > Issue Type: New Feature > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Minor > Fix For: 0.7.0 > > > We want to show the link to Spark UI on the Livy UI while session is in > Running state. > The format of the link url should be configurable. > Examples: > * https://my-cluster.example.com/spark-ui/%s > * [http://%s.my-cluster.example.com/|http://%25s.my-cluster.example.com/] > where %s is Spark appId. > Since Livy UI already displays Spark appId and adds link to it when it is > provided through AppInfo, using it for routing purposes looks right decision. > > This change should unlock the integrations with Kubernetes Ingress resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-703) Show SparkUI link on Livy UI while session is running
Aliaksandr Sasnouskikh created LIVY-703: --- Summary: Show SparkUI link on Livy UI while session is running Key: LIVY-703 URL: https://issues.apache.org/jira/browse/LIVY-703 Project: Livy Issue Type: New Feature Environment: Kubernetes Reporter: Aliaksandr Sasnouskikh Fix For: 0.7.0 We want to show the link to Spark UI on the Livy UI while session is in Running state. The format of the link url should be configurable. Examples: * https://my-cluster.example.com/spark-ui/%s * [http://%s.my-cluster.example.com/|http://%25s.my-cluster.example.com/] where %s is Spark appId. Since Livy UI already displays Spark appId and adds link to it when it is provided through AppInfo, using it for routing purposes looks right decision. This change should unlock the integrations with Kubernetes Ingress resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (LIVY-702) Submit Spark apps to Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-702: Summary: Submit Spark apps to Kubernetes (was: Submit Spark on Kubernetes apps to Kubernetes) > Submit Spark apps to Kubernetes > --- > > Key: LIVY-702 > URL: https://issues.apache.org/jira/browse/LIVY-702 > Project: Livy > Issue Type: New Feature > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > Fix For: 0.7.0 > > > Provide a way to submit Spark apps to Kubernetes. Points to cover: > * Support Batch sessions > * Support Interactive sessions > * Collect logs and diagnostics information > * Restore sessions monitoring after Livy restarts > * GC created Kubernetes resources -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-588) Full support for Spark on Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960443#comment-16960443 ] Aliaksandr Sasnouskikh commented on LIVY-588: - This epic fixes that ticket splitting it to multiple steps: https://issues.apache.org/jira/browse/LIVY-702 > Full support for Spark on Kubernetes > > > Key: LIVY-588 > URL: https://issues.apache.org/jira/browse/LIVY-588 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > > > Since Spark on Kubernetes has been released relatively long ago this can be a > good idea to include Kubernetes support to Livy project as well, as it can > solve much problems related to working with Spark on Kubernetes, it can fully > replace Yarn in case of working atop Kubernetes cluster: > - Livy UI has cached logs/diagnostics page > - Livy UI shows links to Spark UI and Spark History Server > - With Kubernetes Ingress resource Livy can be configured to serve as an > orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support > option to create routes to Spark UI) > - Nginx Ingress solves `basePath` support for Spark UI and History Server as > well as has lots of auth integrations available: > [https://github.com/kubernetes/ingress-nginx] > - Livy UI can be integrated with Grafana Loki logs (PR provides solution for > that) > Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] > Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] > Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 > Design concept: > [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md] > Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes > v1.11.8: > - Image: Spark 2.4.3 with Hadoop 3.2.0 > ([https://github.com/jahstreet/spark-on-kubernetes-docker]) > - History Server: > [https://github.com/helm/charts/tree/master/stable/spark-history-server] > - Jupyter Notebook with Sparkmagic: > [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter] > Unit and/or integration tests are in progress. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (LIVY-702) Submit Spark on Kubernetes apps to Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-702: Description: Provide a way to submit Spark apps to Kubernetes. Points to cover: * Support Batch sessions * Support Interactive sessions * Collect logs and diagnostics information * Restore sessions monitoring after Livy restarts * GC created Kubernetes resources was:Provide a way to submit Spark apps to Kubernetes > Submit Spark on Kubernetes apps to Kubernetes > - > > Key: LIVY-702 > URL: https://issues.apache.org/jira/browse/LIVY-702 > Project: Livy > Issue Type: New Feature > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > Fix For: 0.7.0 > > > Provide a way to submit Spark apps to Kubernetes. Points to cover: > * Support Batch sessions > * Support Interactive sessions > * Collect logs and diagnostics information > * Restore sessions monitoring after Livy restarts > * GC created Kubernetes resources -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-702) Submit Spark on Kubernetes apps to Kubernetes
Aliaksandr Sasnouskikh created LIVY-702: --- Summary: Submit Spark on Kubernetes apps to Kubernetes Key: LIVY-702 URL: https://issues.apache.org/jira/browse/LIVY-702 Project: Livy Issue Type: New Feature Environment: Kubernetes Reporter: Aliaksandr Sasnouskikh Fix For: 0.7.0 Provide a way to submit Spark apps to Kubernetes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-701) Spark on Kubernetes support
Aliaksandr Sasnouskikh created LIVY-701: --- Summary: Spark on Kubernetes support Key: LIVY-701 URL: https://issues.apache.org/jira/browse/LIVY-701 Project: Livy Issue Type: Epic Environment: Kubernetes Reporter: Aliaksandr Sasnouskikh This epic is created to resolve https://issues.apache.org/jira/browse/LIVY-588 by splitting the work to multiple separate steps to ease the review and merge with the master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (LIVY-588) Full support for Spark on Kubernetes
[
https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr Sasnouskikh updated LIVY-588:
Description:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
- Livy UI has cached logs/diagnostics page
- Livy UI shows links to Spark UI and Spark History Server
- With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
- Nginx Ingress solves `basePath` support for Spark UI and History Server as
well as has lots of auth integrations available:
https://github.com/kubernetes/ingress-nginx
- Livy UI can be integrated with Grafana Loki logs (PR provides solution for
that)
Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker
Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm
Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
Design concept:
https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md
## How was this patch tested?
Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes
v1.11.8:
- Image: Spark 2.4.3 with Hadoop 3.2.0
(https://github.com/jahstreet/spark-on-kubernetes-docker)
- History Server:
https://github.com/helm/charts/tree/master/stable/spark-history-server
- Jupyter Notebook with Sparkmagic:
https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter
Unit and/or integration tests are in progress.
was:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
* Livy UI has cached logs/diagnostics page
* Livy UI shows links to Spark UI and Spark History Server
* With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
* Nginx Ingress solves {{basePath}} support for Spark UI and History Server as
well as has lots of auth integrations available:
[https://github.com/kubernetes/ingress-nginx]
Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
Associated PR: [https://github.com/apache/incubator-livy/pull/167]
Design concept doc:
[https://github.com/jahstreet/spark-on-kubernetes-helm/blob/master/README.md]
> Full support for Spark on Kubernetes
>
>
> Key: LIVY-588
> URL: https://issues.apache.org/jira/browse/LIVY-588
> Project: Livy
> Issue Type: New Feature
>Affects Versions: 0.7.0
> Environment: Kubernetes
>Reporter: Aliaksandr Sasnouskikh
>Priority: Major
>
>
> Since Spark on Kubernetes has been released relatively long ago this can be a
> good idea to include Kubernetes support to Livy project as well, as it can
> solve much problems related to working with Spark on Kubernetes, it can fully
> replace Yarn in case of working atop Kubernetes cluster:
> - Livy UI has cached logs/diagnostics page
> - Livy UI shows links to Spark UI and Spark History Server
> - With Kubernetes Ingress resource Livy can be configured to serve as an
> orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
> option to create routes to Spark UI)
> - Nginx Ingress solves `basePath` support for Spark UI and History Server as
> well as has lots of auth integrations available:
> https://github.com/kubernetes/ingress-nginx
> - Livy UI can be integrated with Grafana Loki logs (PR provides solution for
> that)
> Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker
> Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm
> Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
> Design concept:
> https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md
> ## How was this patch tested?
> Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes
> v1.11.8:
> - Image: Spark 2.4.3 with Hadoop 3.2.0
> (https://github.com/jahstreet/spark-on-kubernetes-docker)
> - History Server:
> https://github.com/helm/charts/tree/master/stable/spark-history-server
> - Jupyter Notebook with Sparkmagic:
> https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter
> Unit and/or integration tests are in progress.
--
This
[jira] [Updated] (LIVY-588) Full support for Spark on Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-588: Description: Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster: - Livy UI has cached logs/diagnostics page - Livy UI shows links to Spark UI and Spark History Server - With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI) - Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: [https://github.com/kubernetes/ingress-nginx] - Livy UI can be integrated with Grafana Loki logs (PR provides solution for that) Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 Design concept: [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md] Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8: - Image: Spark 2.4.3 with Hadoop 3.2.0 ([https://github.com/jahstreet/spark-on-kubernetes-docker]) - History Server: [https://github.com/helm/charts/tree/master/stable/spark-history-server] - Jupyter Notebook with Sparkmagic: [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter] Unit and/or integration tests are in progress. was: Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster: - Livy UI has cached logs/diagnostics page - Livy UI shows links to Spark UI and Spark History Server - With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI) - Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: https://github.com/kubernetes/ingress-nginx - Livy UI can be integrated with Grafana Loki logs (PR provides solution for that) Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 Design concept: https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md ## How was this patch tested? Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8: - Image: Spark 2.4.3 with Hadoop 3.2.0 (https://github.com/jahstreet/spark-on-kubernetes-docker) - History Server: https://github.com/helm/charts/tree/master/stable/spark-history-server - Jupyter Notebook with Sparkmagic: https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter Unit and/or integration tests are in progress. > Full support for Spark on Kubernetes > > > Key: LIVY-588 > URL: https://issues.apache.org/jira/browse/LIVY-588 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > > > Since Spark on Kubernetes has been released relatively long ago this can be a > good idea to include Kubernetes support to Livy project as well, as it can > solve much problems related to working with Spark on Kubernetes, it can fully > replace Yarn in case of working atop Kubernetes cluster: > - Livy UI has cached logs/diagnostics page > - Livy UI shows links to Spark UI and Spark History Server > - With Kubernetes Ingress resource Livy can be configured to serve as an > orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support > option to create routes to Spark UI) > - Nginx Ingress solves `basePath` support for Spark UI and History Server as > well as has lots of auth integrations available: > [https://github.com/kubernetes/ingress-nginx] > - Livy UI can be integrated with Grafana Loki logs (PR provides solution for > that) > Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] > Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] > Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 > Design concept: >
[jira] [Created] (LIVY-588) Full support for Spark on Kubernetes
Aliaksandr Sasnouskikh created LIVY-588:
---
Summary: Full support for Spark on Kubernetes
Key: LIVY-588
URL: https://issues.apache.org/jira/browse/LIVY-588
Project: Livy
Issue Type: New Feature
Affects Versions: 0.7.0
Environment: Kubernetes
Reporter: Aliaksandr Sasnouskikh
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
* Livy UI has cached logs/diagnostics page
* Livy UI shows links to Spark UI and Spark History Server
* With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
* Nginx Ingress solves {{basePath}} support for Spark UI and History Server as
well as has lots of auth integrations available:
[https://github.com/kubernetes/ingress-nginx]
Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
Associated PR: [https://github.com/apache/incubator-livy/pull/167]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
