[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637602#comment-15637602 ] Anirudh Ramanathan commented on SPARK-18278: Corresponding issue in kubernetes:

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718478#comment-15718478 ] Anirudh Ramanathan commented on SPARK-18278: There is a way to use a standard image that

[jira] [Updated] (SPARK-18662) Move cluster managers into their own sub-directory

[ https://issues.apache.org/jira/browse/SPARK-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-18662: --- External issue URL: https://github.com/apache/spark/pull/16092 > Move cluster

[jira] [Created] (SPARK-18662) Move cluster managers into their own sub-directory

Anirudh Ramanathan created SPARK-18662: -- Summary: Move cluster managers into their own sub-directory Key: SPARK-18662 URL: https://issues.apache.org/jira/browse/SPARK-18662 Project: Spark

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111546#comment-16111546 ] Anirudh Ramanathan commented on SPARK-18278: FYI: sandflee's question was addressed in

[jira] [Updated] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-18278: --- Summary: SPIP: Support native submission of spark jobs to a kubernetes cluster

[jira] [Comment Edited] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111546#comment-16111546 ] Anirudh Ramanathan edited comment on SPARK-18278 at 8/2/17 7:11 PM:

[jira] [Updated] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-18278: --- Attachment: SPARK-18278 Spark on Kubernetes Design Proposal Revision 2 (1).pdf

[jira] [Commented] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158100#comment-16158100 ] Anirudh Ramanathan commented on SPARK-18278: The plan is to support k8s features (beta/GA) in

[jira] [Commented] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183960#comment-16183960 ] Anirudh Ramanathan commented on SPARK-18278: An update on the plan - 2 days ago, we cut

[jira] [Comment Edited] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183960#comment-16183960 ] Anirudh Ramanathan edited comment on SPARK-18278 at 9/28/17 10:17 AM:

[jira] [Updated] (SPARK-22646) Spark on Kubernetes - basic submission client

[ https://issues.apache.org/jira/browse/SPARK-22646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22646: --- Description: The submission client is responsible for creating the Kubernetes pod

[jira] [Created] (SPARK-22648) Documentation for Kubernetes Scheduler Backend

Anirudh Ramanathan created SPARK-22648: -- Summary: Documentation for Kubernetes Scheduler Backend Key: SPARK-22648 URL: https://issues.apache.org/jira/browse/SPARK-22648 Project: Spark

[jira] [Created] (SPARK-22647) Docker files for image creation

Anirudh Ramanathan created SPARK-22647: -- Summary: Docker files for image creation Key: SPARK-22647 URL: https://issues.apache.org/jira/browse/SPARK-22647 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-22645) Add Kubernetes Scheduler Backend

[ https://issues.apache.org/jira/browse/SPARK-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-22645. Resolution: Fixed Fixed by https://github.com/apache/spark/pull/19468 > Add

[jira] [Created] (SPARK-22645) Add Kubernetes Scheduler Backend

Anirudh Ramanathan created SPARK-22645: -- Summary: Add Kubernetes Scheduler Backend Key: SPARK-22645 URL: https://issues.apache.org/jira/browse/SPARK-22645 Project: Spark Issue Type:

[jira] [Updated] (SPARK-22645) Add Scheduler Backend with static allocation of executors

[ https://issues.apache.org/jira/browse/SPARK-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22645: --- Summary: Add Scheduler Backend with static allocation of executors (was: Add

[jira] [Created] (SPARK-22646) Spark on Kubernetes - basic submission client

Anirudh Ramanathan created SPARK-22646: -- Summary: Spark on Kubernetes - basic submission client Key: SPARK-22646 URL: https://issues.apache.org/jira/browse/SPARK-22646 Project: Spark

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294759#comment-16294759 ] Anirudh Ramanathan commented on SPARK-22647: [~wangyang] libc6-compat is in there now. Can

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292113#comment-16292113 ] Anirudh Ramanathan commented on SPARK-22647: I think we haven't run that particular

[jira] [Updated] (SPARK-22807) Change configuration options to use "container" instead of "docker"

[ https://issues.apache.org/jira/browse/SPARK-22807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22807: --- Summary: Change configuration options to use "container" instead of "docker" (was:

[jira] [Created] (SPARK-22807) Change commandline options to use "container" instead of "docker"

Anirudh Ramanathan created SPARK-22807: -- Summary: Change commandline options to use "container" instead of "docker" Key: SPARK-22807 URL: https://issues.apache.org/jira/browse/SPARK-22807

[jira] [Updated] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22777: --- [~liyinan926] [~ssuchter] > Docker container built for Kubernetes doesn't allow running

[jira] [Updated] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22777: --- Description: Default docker images built thrown an error when trying to run on a

[jira] [Updated] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22777: --- Description: Default docker images built thrown an error when trying to run on a

[jira] [Updated] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22777: --- Description: Default docker images built thrown an error when trying to run on a

[jira] [Updated] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22777: --- Description: Default docker images built thrown an error when trying to run on a

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289805#comment-16289805 ] Anirudh Ramanathan commented on SPARK-22647: [~mcheah] [~erikerlandson] PTAL > Docker files

[jira] [Updated] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22778: --- Description: Building images based on master and deploying Spark PI results in the

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289818#comment-16289818 ] Anirudh Ramanathan commented on SPARK-22778: I submitted it as `k8s://https://xx.yy.zz.ww` to

[jira] [Updated] (SPARK-22842) Support consuming oauth token and certs through environment variables

[ https://issues.apache.org/jira/browse/SPARK-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22842: --- Priority: Minor (was: Critical) > Support consuming oauth token and certs through

[jira] [Created] (SPARK-22842) Support consuming oauth token and certs through environment variables

Anirudh Ramanathan created SPARK-22842: -- Summary: Support consuming oauth token and certs through environment variables Key: SPARK-22842 URL: https://issues.apache.org/jira/browse/SPARK-22842

[jira] [Created] (SPARK-22845) Modify spark.kubernetes.allocation.batch.delay to take time instead of int

Anirudh Ramanathan created SPARK-22845: -- Summary: Modify spark.kubernetes.allocation.batch.delay to take time instead of int Key: SPARK-22845 URL: https://issues.apache.org/jira/browse/SPARK-22845

[jira] [Created] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

Anirudh Ramanathan created SPARK-22777: -- Summary: Docker container built for Kubernetes doesn't allow running entrypoint.sh Key: SPARK-22777 URL: https://issues.apache.org/jira/browse/SPARK-22777

[jira] [Updated] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully.

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22778: --- Summary: Kubernetes scheduler at master failing to run applications successfully.

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing tests

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289808#comment-16289808 ] Anirudh Ramanathan commented on SPARK-22778: [~mcheah] [~kimoonkim] [~ifilonenko] PTAL >

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289881#comment-16289881 ] Anirudh Ramanathan commented on SPARK-22778: Excellent - thanks for the quick debug Matt. Now

[jira] [Comment Edited] (SPARK-22777) Docker container built for Kubernetes doesn't allow running entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-22777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289795#comment-16289795 ] Anirudh Ramanathan edited comment on SPARK-22777 at 12/13/17 8:02 PM:

[jira] [Updated] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22778: --- Summary: Kubernetes scheduler at master failing to run applications successfully

[jira] [Created] (SPARK-22778) Kubernetes scheduler at master failing tests

Anirudh Ramanathan created SPARK-22778: -- Summary: Kubernetes scheduler at master failing tests Key: SPARK-22778 URL: https://issues.apache.org/jira/browse/SPARK-22778 Project: Spark

[jira] [Comment Edited] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289818#comment-16289818 ] Anirudh Ramanathan edited comment on SPARK-22778 at 12/13/17 8:23 PM:

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289868#comment-16289868 ] Anirudh Ramanathan commented on SPARK-22778: I've verified that the image contains the right

[jira] [Comment Edited] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289868#comment-16289868 ] Anirudh Ramanathan edited comment on SPARK-22778 at 12/13/17 8:41 PM:

[jira] [Updated] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24179: --- Issue Type: New Feature (was: Task) > History Server for Kubernetes >

[jira] [Commented] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474772#comment-16474772 ] Anirudh Ramanathan commented on SPARK-24249: cc/ [~mccheah] > Spark on kubernetes, pods

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474770#comment-16474770 ] Anirudh Ramanathan commented on SPARK-24266: Thanks for filing this issue and digging in. It

[jira] [Comment Edited] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474774#comment-16474774 ] Anirudh Ramanathan edited comment on SPARK-24248 at 5/14/18 8:44 PM: -

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474774#comment-16474774 ] Anirudh Ramanathan commented on SPARK-24248: Somewhat related -

[jira] [Assigned] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24137: -- Assignee: Matt Cheah > [K8s] Mount temporary directories in emptydir volumes

[jira] [Resolved] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24137. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21238

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496962#comment-16496962 ] Anirudh Ramanathan commented on SPARK-24434: The way several custom APIs have done this

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan commented on SPARK-24434: Good point. I was basing my suggestion of JSON

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan edited comment on SPARK-24434 at 5/31/18 6:54 PM: -

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497014#comment-16497014 ] Anirudh Ramanathan commented on SPARK-24434: Open to suggestions on what could be intuitive

[jira] [Assigned] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24232: -- Assignee: Stavros Kontopoulos > Allow referring to kubernetes secrets as env

[jira] [Resolved] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24232. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request

[jira] [Resolved] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24547. Resolution: Fixed Fix Version/s: 2.4.0 > Spark on K8s docker-image-tool.sh

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24547: -- Assignee: Anirudh Ramanathan > Spark on K8s docker-image-tool.sh

[jira] [Commented] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518738#comment-16518738 ] Anirudh Ramanathan commented on SPARK-24547: Resolved by

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24547: -- Assignee: (was: Anirudh Ramanathan) > Spark on K8s docker-image-tool.sh

[jira] [Created] (SPARK-24600) Improve support for building subset of images in dockerfile

Anirudh Ramanathan created SPARK-24600: -- Summary: Improve support for building subset of images in dockerfile Key: SPARK-24600 URL: https://issues.apache.org/jira/browse/SPARK-24600 Project:

[jira] [Updated] (SPARK-24600) Improve support for building different types of images in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-24600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24600: --- Description: Our docker images currently build and push docker images for pyspark

[jira] [Updated] (SPARK-24600) Improve support for building different types of images in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-24600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24600: --- Summary: Improve support for building different types of images in dockerfile

[jira] [Resolved] (SPARK-24428) Remove unused code and fix any related doc in K8s module

[ https://issues.apache.org/jira/browse/SPARK-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24428. Resolution: Fixed Fix Version/s: 2.4.0 > Remove unused code and fix any

[jira] [Commented] (SPARK-24428) Remove unused code and fix any related doc in K8s module

[ https://issues.apache.org/jira/browse/SPARK-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530391#comment-16530391 ] Anirudh Ramanathan commented on SPARK-24428: Resolved by

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460559#comment-16460559 ] Anirudh Ramanathan commented on SPARK-24135: +1 to detecting all pod error states and doing

[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460554#comment-16460554 ] Anirudh Ramanathan commented on SPARK-24137: SGTM. Let's try and share as much code as

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan edited comment on SPARK-24135 at 5/3/18 8:58 AM:

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan commented on SPARK-24135: It is increasingly common for people to write

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan edited comment on SPARK-24135 at 5/3/18 9:01 AM:

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462145#comment-16462145 ] Anirudh Ramanathan commented on SPARK-24135: cc/ [~mridulm80] [~irashid] for thoughts on

[jira] [Created] (SPARK-22866) Kubernetes dockerfile path needs update

Anirudh Ramanathan created SPARK-22866: -- Summary: Kubernetes dockerfile path needs update Key: SPARK-22866 URL: https://issues.apache.org/jira/browse/SPARK-22866 Project: Spark Issue

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300810#comment-16300810 ] Anirudh Ramanathan commented on SPARK-22647: The dockerfiles themselves have been checked in

[jira] [Resolved] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-22647. Resolution: Fixed > Docker files for image creation >

[jira] [Created] (SPARK-22865) Publish Official Apache Spark Docker images

Anirudh Ramanathan created SPARK-22865: -- Summary: Publish Official Apache Spark Docker images Key: SPARK-22865 URL: https://issues.apache.org/jira/browse/SPARK-22865 Project: Spark

[jira] [Created] (SPARK-23010) Add integration

Anirudh Ramanathan created SPARK-23010: -- Summary: Add integration Key: SPARK-23010 URL: https://issues.apache.org/jira/browse/SPARK-23010 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23010: --- Summary: Add integration testing for Kubernetes backend into the apache/spark

[jira] [Commented] (SPARK-22853) K8s or Kubernetes?

[ https://issues.apache.org/jira/browse/SPARK-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319119#comment-16319119 ] Anirudh Ramanathan commented on SPARK-22853: It turns out this was a Kubernetes community

[jira] [Created] (SPARK-23016) Spark UI access and documentation

Anirudh Ramanathan created SPARK-23016: -- Summary: Spark UI access and documentation Key: SPARK-23016 URL: https://issues.apache.org/jira/browse/SPARK-23016 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319412#comment-16319412 ] Anirudh Ramanathan commented on SPARK-23016: cc/ [~mccheah] > Spark UI access and

[jira] [Created] (SPARK-22992) Remove assumption of cluster domain in Kubernetes mode

Anirudh Ramanathan created SPARK-22992: -- Summary: Remove assumption of cluster domain in Kubernetes mode Key: SPARK-22992 URL: https://issues.apache.org/jira/browse/SPARK-22992 Project: Spark

[jira] [Updated] (SPARK-23146) Support client mode for Kubernetes cluster backend

[ https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23146: --- Target Version/s: 2.4.0 > Support client mode for Kubernetes cluster backend >

[jira] [Commented] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331153#comment-16331153 ] Anirudh Ramanathan commented on SPARK-22962: I think this isn't super critical for this

[jira] [Commented] (SPARK-23133) Spark options are not passed to the Executor in Docker context

[ https://issues.apache.org/jira/browse/SPARK-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331074#comment-16331074 ] Anirudh Ramanathan commented on SPARK-23133: Thanks for submitting the PR fixing this. >

[jira] [Commented] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331081#comment-16331081 ] Anirudh Ramanathan commented on SPARK-23082: This is an interesting feature request. We had

[jira] [Commented] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331085#comment-16331085 ] Anirudh Ramanathan commented on SPARK-22962: This is the resource staging server use-case. 

[jira] [Updated] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22962: --- Affects Version/s: (was: 2.3.0) 2.4.0 > Kubernetes app

[jira] [Updated] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22962: --- Affects Version/s: (was: 2.4.0) 2.3.0 > Kubernetes app

[jira] [Updated] (SPARK-23146) Support client mode for Kubernetes cluster backend

[ https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23146: --- Affects Version/s: (was: 2.4.0) 2.3.0 > Support client

[jira] [Comment Edited] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331081#comment-16331081 ] Anirudh Ramanathan edited comment on SPARK-23082 at 1/18/18 7:52 PM: -

[jira] [Resolved] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23016. Resolution: Fixed This is resolved and we've verified it for 2.3.0. > Spark UI

[jira] [Updated] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-19700: --- Issue Type: Improvement (was: Bug) > Design an API for pluggable scheduler

[jira] [Commented] (SPARK-23083) Adding Kubernetes as an option to https://spark.apache.org/

[ https://issues.apache.org/jira/browse/SPARK-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326631#comment-16326631 ] Anirudh Ramanathan commented on SPARK-23083: Thanks. I'll create a PR against that repo.

[jira] [Created] (SPARK-23083) Adding Kubernetes as an option to https://spark.apache.org/

Anirudh Ramanathan created SPARK-23083: -- Summary: Adding Kubernetes as an option to https://spark.apache.org/ Key: SPARK-23083 URL: https://issues.apache.org/jira/browse/SPARK-23083 Project:

[jira] [Commented] (SPARK-23083) Adding Kubernetes as an option to https://spark.apache.org/

[ https://issues.apache.org/jira/browse/SPARK-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326711#comment-16326711 ] Anirudh Ramanathan commented on SPARK-23083: opened

[jira] [Updated] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23016: --- Priority: Minor (was: Major) > Spark UI access and documentation >

[jira] [Commented] (SPARK-23104) Document that kubernetes is still "experimental"

[ https://issues.apache.org/jira/browse/SPARK-23104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329608#comment-16329608 ] Anirudh Ramanathan commented on SPARK-23104: Thanks for opening this - is this simply a

[jira] [Commented] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329675#comment-16329675 ] Anirudh Ramanathan commented on SPARK-23016: Works as expected with port forwarding in my

[jira] [Commented] (SPARK-23104) Document that kubernetes is still "experimental"

[ https://issues.apache.org/jira/browse/SPARK-23104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329621#comment-16329621 ] Anirudh Ramanathan commented on SPARK-23104: [~sameerag], this is a docs-only change, will

[jira] [Created] (SPARK-23137) spark.kubernetes.executor.podNamePrefix is ignored

Anirudh Ramanathan created SPARK-23137: -- Summary: spark.kubernetes.executor.podNamePrefix is ignored Key: SPARK-23137 URL: https://issues.apache.org/jira/browse/SPARK-23137 Project: Spark

  1   2   >