[jira] [Updated] (LIVY-702) Submit Spark apps to Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-702: Summary: Submit Spark apps to Kubernetes (was: Submit Spark on Kubernetes apps to Kubernetes) > Submit Spark apps to Kubernetes > --- > > Key: LIVY-702 > URL: https://issues.apache.org/jira/browse/LIVY-702 > Project: Livy > Issue Type: New Feature > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > Fix For: 0.7.0 > > > Provide a way to submit Spark apps to Kubernetes. Points to cover: > * Support Batch sessions > * Support Interactive sessions > * Collect logs and diagnostics information > * Restore sessions monitoring after Livy restarts > * GC created Kubernetes resources -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-588) Full support for Spark on Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960443#comment-16960443 ] Aliaksandr Sasnouskikh commented on LIVY-588: - This epic fixes that ticket splitting it to multiple steps: https://issues.apache.org/jira/browse/LIVY-702 > Full support for Spark on Kubernetes > > > Key: LIVY-588 > URL: https://issues.apache.org/jira/browse/LIVY-588 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > > > Since Spark on Kubernetes has been released relatively long ago this can be a > good idea to include Kubernetes support to Livy project as well, as it can > solve much problems related to working with Spark on Kubernetes, it can fully > replace Yarn in case of working atop Kubernetes cluster: > - Livy UI has cached logs/diagnostics page > - Livy UI shows links to Spark UI and Spark History Server > - With Kubernetes Ingress resource Livy can be configured to serve as an > orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support > option to create routes to Spark UI) > - Nginx Ingress solves `basePath` support for Spark UI and History Server as > well as has lots of auth integrations available: > [https://github.com/kubernetes/ingress-nginx] > - Livy UI can be integrated with Grafana Loki logs (PR provides solution for > that) > Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] > Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] > Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 > Design concept: > [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md] > Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes > v1.11.8: > - Image: Spark 2.4.3 with Hadoop 3.2.0 > ([https://github.com/jahstreet/spark-on-kubernetes-docker]) > - History Server: > [https://github.com/helm/charts/tree/master/stable/spark-history-server] > - Jupyter Notebook with Sparkmagic: > [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter] > Unit and/or integration tests are in progress. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (LIVY-702) Submit Spark on Kubernetes apps to Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-702: Description: Provide a way to submit Spark apps to Kubernetes. Points to cover: * Support Batch sessions * Support Interactive sessions * Collect logs and diagnostics information * Restore sessions monitoring after Livy restarts * GC created Kubernetes resources was:Provide a way to submit Spark apps to Kubernetes > Submit Spark on Kubernetes apps to Kubernetes > - > > Key: LIVY-702 > URL: https://issues.apache.org/jira/browse/LIVY-702 > Project: Livy > Issue Type: New Feature > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > Fix For: 0.7.0 > > > Provide a way to submit Spark apps to Kubernetes. Points to cover: > * Support Batch sessions > * Support Interactive sessions > * Collect logs and diagnostics information > * Restore sessions monitoring after Livy restarts > * GC created Kubernetes resources -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-702) Submit Spark on Kubernetes apps to Kubernetes
Aliaksandr Sasnouskikh created LIVY-702: --- Summary: Submit Spark on Kubernetes apps to Kubernetes Key: LIVY-702 URL: https://issues.apache.org/jira/browse/LIVY-702 Project: Livy Issue Type: New Feature Environment: Kubernetes Reporter: Aliaksandr Sasnouskikh Fix For: 0.7.0 Provide a way to submit Spark apps to Kubernetes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-701) Spark on Kubernetes support
Aliaksandr Sasnouskikh created LIVY-701: --- Summary: Spark on Kubernetes support Key: LIVY-701 URL: https://issues.apache.org/jira/browse/LIVY-701 Project: Livy Issue Type: Epic Environment: Kubernetes Reporter: Aliaksandr Sasnouskikh This epic is created to resolve https://issues.apache.org/jira/browse/LIVY-588 by splitting the work to multiple separate steps to ease the review and merge with the master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
